‹ Retour

 

Internal Knowledge Base Assistant

 

By Vladimir Williams

18/09/2026

 

So you have a large, proprietary knowledge base (KB) of documentation and want an AI assistant to help in retrieving and synthesizing information while maintaining privacy and traceability. That is what retrieval augmented generation (RAG) is for. This is not an AI model which needs to be trained on your source documents. Rather, a RAG is a pipeline which processes information from a defined KB to allow an LLM to synthesize from the information therein. It can be implemented completely locally and the LLM can respond to frequent KB updates—there's no need to risk sending confidential information to a third party.

 

 

1. Chunking & Embedding

First, the documents in the KB are broken up into overlapping chunks—segments of the texts, large enough to capture the contained context. These plain-text chunks are then embedded into a high-dimensional semantic space using a small AI model—I used bge-small-en-v1.5. In essence, these chunks are high-dimensional vectors. Chunks near each other in this semantic space are more related than chunks which are further.

 

2. Vector Indexing

Embedded chunks are then indexed to fascilitate a similarity search later on. When the user queries the RAG, their query will be embedded into that high-dimensional semantic space too and the indexing will help identify nearby chunks to collect information from. 

 

3. LLM Synthesis

Finally, the user query is embedded into the same semantic space as the previously-indexed KB chunks and the top k most-similar chunk vectors are formed into a prompt for and LLM to process—I used Mistral-7B-v0.3, a ~4GB model which runs fine on my home PC. The value of k should be tuned carefully: if you set it too low, there's a higher chance of the LLM missing the relevant passage in the KB, but if it's too high, the LLM takes longer to generate an answer accurate to the KB. In practice, a value of k between 3 and 10 is used, depending on the size of the KB.

 

A typical workflow then looks like this:

 

Sample Execution and Testing

 

I designed this internal KB assistant to print a manifest of the internal knowledge base: how many documents are contained and how many chunks per document. A more detailed manifest may include number of lines per document, publication date, authors, etc. basically metadata on each document to answer questions the user might have about the KB itself, like "What's the most recently-published document in the KB?"

 

 

When we can ask questions about the content in the KB, the assistant gives a response with references:

 

 

The assistant lets us know when questions go beyond the limits of the KB:

 

 

The assistant synthesizes information from related content outside the KB, but does not share the KB:

 

 

Further refinement is, of course, possible.

 

Follow the whole project on GitHub!