How do I create and manage knowledge bases?

Overview  

A knowledge base is a collection of documents your model can refer to during your chat conversations. Think of this like a limited memory from which the model can pull information, making responses more contextually aware. These user-curated documents can come in a variety of formats, but must contain machine-readable text. You can create, edit, and directly share content from your knowledge base.

After every user message, relevant sections from the knowledge base are retrieved and passed to the model as context. While the knowledge base will serve to inform model responses, there are some limitations to consider.

Adding a Collection  

To create a knowledge base, you must first create a named collection for documents before you can upload them.

To add a collection:

• Navigate to the Workspace Knowledge area (Choose “Workspace” from the left sidebar and select “Knowledge”)
• Click the “+” sign to add a new collection

Creating a Collection  

From the resulting screen, enter a name and description for your collection. By default, the visibility of your collection is set as Private. If you subsequently select a Group or Groups, you can set up Read/Write permissions by Group to allow for collaboration in managing the collection.

Adding Content to the Collection  

Content must contain text in order to be useful in this context, and the application supports multiple formats (e.g., Microsoft Word, Adobe Acrobat PDF, text files, etc.). Of note, keep in mind that images or documents scanned to PDF might not contain machine-readable text. You can upload individual files or upload (and even sync) multiple files from a directory accessible from your local computer. Further, you can write or copy/paste text directly into the collection using the “Add text content” link.

Reviewing Content  

Once a file or files have been uploaded, you can then review the text that has been extracted. If you have uploaded from a pre-formatted document (e.g., a PDF), you may find that there are extra spaces or characters, or that headings and text are out of order.

We recommend spending time to correct any visible errors from this screen, as the better the content inputs, the better the eventual responses from the Model.

Using your knowledge base  

Once you have set up your knowledge base, you can manage it from Workspace.

You can actively use it from any prompt by typing “#” into the model prompt. This will provide a list of knowledge base collections from which you can choose the appropriate collection to inform your response.

You can also attach your knowledge base to a custom model. When setting up your custom model, select your knowledge base under “Knowledge”.

Knowledge Base Limitations  

Of note, while the knowledge base will inform the results of your search, what is passed to the model is limited to a certain number of tokens. Be thoughtful of this limitation when creating and managing your knowledge base.

Didn’t find what you needed? Please reach out to research.computing@dartmouth.edu.