How to improve the accuracy of a large model private knowledge base? Slicing is the key

Preface

RAG is like an open-book exam. Before answering the questions, we first look through the materials we prepared (enhance our abilities by searching for relevant documents). This way, the answers will not only be more accurate, but also include more contextual information, making them more in-depth and targeted.

In RAG, we need to prepare knowledge materials in advance, then vectorize them and store them in a vector database. When answering questions, we vectorize the questions and then search our database to see if there is similar content. If so, we recall the content and give it to a large model (such as Deepseek). Then, we give the corresponding answers through the reasoning and induction of the large model.

Similar to the following steps:

Step 1: Prepare data   First, we need to “cut” the prepared knowledge documents into small document blocks and store them in the database. If these small document blocks also generate corresponding “embeddings” (which can be understood as a mathematical representation), then the database can be upgraded to vector storage to facilitate subsequent fast search.

Step 2: Retrieval   When asking a question, the system first vectorizes the question and searches the database for the most relevant small document blocks through vector search, full-text search, or multi-modal methods. After locating the information fragments that best match the user's question, they are fed to the big model in a contextual manner. This way, the big model can not only find the answer faster, but also ensure that the answer is more accurate and targeted, while also reducing illusions.

Challenges 

For enterprises or individuals, there are many policy articles or documents, which are usually very long. If these long articles are given to a large model, the existing large model's computing power cannot process them all at once, so the text needs to be cut into pieces. However, the bigger the piece, the better. When vectorizing, long texts face the following core challenges:

1. Semantic Information Dilution

• 1. Multi-topic coverage : Long texts often contain multiple topics or viewpoints. The overall vector is difficult to accurately capture the semantic details, resulting in the core content being diluted.
• 2. Loss of contextual continuity : Directly averaging word vectors or using [CLS] tags cannot effectively handle the coherent semantic relationship of long texts, which may break the contextual logic.

2. Computational Complexity and Resource Consumption

• 1. Attention mechanism bottleneck : The self-attention layer of the Transformer architecture has O(n²) complexity. When processing long texts of more than a thousand words, the computing resources grow exponentially.
• 2. Memory pressure : The generated high-dimensional vector requires a lot of storage space, which may reduce the matching efficiency during retrieval.

3. Limitations of Text Blocking Technology

• 1. Optimal block size selection : Too large blocks lead to semantic confusion, too small blocks destroy context coherence, 50% overlapping blocks can alleviate this problem but increase storage redundancy.
• 2. Special content processing : The segmentation of non-continuous text such as tables and code snippets may destroy the structured information and affect the subsequent QA effect.

4. Model Architecture Constraints

• 1. Input length limit : The maximum input length of most pre-trained models is between 512-2048 tokens. The excess length needs to be truncated or segmented.
• 2. Position encoding distortion : Block processing will reset the position encoding, affecting the model's capture of long-range dependencies.

5. Balance between quality and efficiency

• 1. Dimension optimization dilemma : high-dimensional vectors improve accuracy but increase computational cost, while low-dimensional vectors simplify computation but lose semantic information.
• 2. Noise sensitivity : The impact of noise such as spelling errors and colloquial expressions on vector quality in long texts is amplified.

Slicing Methodology 

Above we can see that "chunking" plays a key role and faces challenges in RAG. It directly determines the accuracy of our recalled knowledge, so it is particularly important to choose an appropriate chunking method; the key to effectively splitting files is to balance the integrity of information and the convenience of management. You can adopt a variety of strategies such as fixed-size chunking, semantic chunking, recursive chunking, chunking based on document structure, and chunking based on LLM .

• 1. Fixed-size chunking : Split the text into chunks based on a predefined number of characters, words, or tokens while retaining a certain amount of overlap. This method is simple to implement, but may truncate sentences, causing information to be scattered across different chunks. 1
• 2. Semantic Chunking : Segment the document based on meaningful units, and continue to add units to existing chunks until the cosine similarity drops significantly. This approach maintains the natural flow of language.
• 3. ‌Recursive Chunking‌ : Chunking is done based on intrinsic delimiters such as paragraphs or chapters. If a chunk exceeds the limit, it is further split into smaller chunks. This approach also maintains the natural flow of the language.
• 4. Chunking based on document structure : Chunking is done by using the intrinsic structure of the document (such as titles, chapters, or paragraphs). This method can maintain the natural structure of the document, but the premise is that the document has a clear structure.
• LLM-based chunking : Use hint engineering to guide LLM to generate meaningful chunking. This approach incorporates the intelligence of large models but may require more computational resources and time.

The summary table is as follows:

Slicing methods supported by RAGFlow 

It is a relatively good way to choose a suitable block strategy based on business scenarios and text characteristics. RAGFlow supports multiple block methods. The following table shows the descriptions of different block methods and their supported document formats.

Chunking method: