AI Search and Vector Data-How do models encode information and data into knowledge?

For each concept and knowledge point, we provide a more in-depth and detailed analysis and description based on its application principles, specific application scenarios, and application methods in practice.

• 
  

  Unstructured Data Objects:

  
  

• 
  

  Internal documents:

   Meeting minutes, reports, emails, chat logs, etc.

  
  

• 
  

  Internet content:  news articles, blog posts, social media updates, user comments.

  
  

• 
  

  Multimedia materials:  product photo gallery, surveillance video clips, and audio files uploaded by users.

  
  

• 
  

  Application principle:

   Traditional databases rely on structured schemas for querying and cannot effectively process text, images, audio, and video data with diverse formats and no fixed structure. Vector embedding technology can convert these objects into a unified numerical representation (vector), making them computable and comparable.

  
  

  
  

  
  

  
  

• 
  

  Application method:

   First, you need to identify the data type, then select the appropriate tool or process (such as OCR, text extraction library, audio and video transcription tool) to convert it into a processable format (such as plain text, image file), and then feed it into the embedding model.

  
  

• 
  

  Inverted Indexes:

  
  

• 
  

  Application principle:

   The core is a data structure, which usually consists of two parts: a term dictionary and a postings list. The dictionary stores the terms (or other index units) that appear in all documents and points to their postings lists. The postings list records the document ID, term frequency, position, and other information containing the term. When querying, first search for the query term in the dictionary, obtain the corresponding postings list, and then find matching documents by merging (such as AND operation to obtain intersection, OR operation to obtain union) the postings lists of multiple terms.

  
  

• 
  

  Application scenarios:  the core of full-text search engines (such as Elasticsearch, Lucene); the text search function of the database; traditional information retrieval systems.

  
  

• 
  

  Application method:  When creating an index in a search engine or a database that supports full-text search, the system will automatically build an inverted index for the specified text field. Users enter keywords through the query interface to search.

  
  

• 
  

  Sharding:

  
  

• 
  

  Application principle:

   A form of horizontal partitioning that distributes the rows (or documents) of a database (table or collection) across multiple independent database instances (shards). A shard key is used to determine which shard a row belongs to. When querying, the routing component (such as mongos in MongoDB) directs the request to the corresponding shard based on the shard key in the query condition, or broadcasts the request to all shards and aggregates the results when it cannot be determined.

  
  

• 
  

  Application scenarios:  Processing large-scale data sets that exceed the capacity or performance limit of a single machine; improving the concurrency of writing and reading; improving system availability (a shard failure does not affect other shards). Widely used in large relational databases, NoSQL databases, and vector databases.

  
  

• 
  

  Application method:  Configure at the database level. You need to select a suitable sharding key (which affects data distribution balance and query efficiency) and determine the sharding strategy (such as hash sharding, range sharding). The database cluster usually automatically handles data distribution and query routing. For example, in Milvus, you can configure the number of shards ( shard_num ) of a Collection.

  
  

• 
  

  Multi Tenancy:

  
  

• 
  

  Application principle:

   On shared infrastructure (hardware, database instances), multiple customers (tenants) are served through logical isolation at the software level (such as using tenant IDs to distinguish data, independent access control policies, and resource quota limits). Each tenant feels like they are using an independent system, but in fact they share resources, thereby reducing costs.

  
  

• 
  

  Application scenarios:  SaaS (Software as a Service) applications, such as CRM systems, project management tools, cloud database services (including many vector database cloud services).

  
  

• 
  

  Application method:  For service providers, it is necessary to design a tenant isolation mechanism at the application architecture level. For users, they usually obtain an isolated tenant environment when registering for a service, access it through an API Key or a specific account, and do not need to care about the underlying sharing details.

  
  

• 
  

  Relational Database:

  
  

• 
  

  Application principle:

   Based on the relational model, data is stored in tables with fixed columns (fields) and data types, and rows (records) represent instances. Entity integrity and referential integrity are enforced through primary keys and foreign keys. SQL is used for data definition, operation, and query, supporting ACID transactions (atomicity, consistency, isolation, and persistence) to ensure data consistency and reliability.

  
  

• 
  

  Application scenarios:  transaction processing systems (OLTP), business data management, scenarios that require strong consistency guarantees, and storage of structured data. For example, order management, user accounts, and inventory systems.

  
  

• 
  

  Application method:  Design the database schema, create the table structure, and use SQL statements to perform add, delete, modify, and query operations. Query performance can be optimized through indexes (such as B-Tree indexes).

  
  

• 
  

  PQ (Product Quantization):

  
  

• 
  

  Application principle:

   A vector lossy compression technique designed to reduce storage space and speed up distance calculations.

  
  

1. 
   

   segmentation:

    Split the original high-dimensional vector (say, D dimensions) into M low-dimensional sub-vectors (each of D/M dimensions).

   
   

2. 
   

   Clustering:  Cluster the i-th subvector of all vectors in the data set (usually using K-Means) to obtain K cluster centers (codewords) to form a codebook for the i-th subspace.

   
   

3. 
   

   Quantization:  For any original vector, replace each of its subvectors with the ID of the closest codeword in the subspace codebook to which it belongs.

   
   

4. 
   

   Storage:  The original vector is compressed into a short code consisting of M codeword IDs.

   
   

5. 
   

   Distance calculation:  The distance between compressed vectors can be quickly estimated by pre-calculating the distance between codewords or using approximate distances (such as asymmetric distance calculation ADC).

   
   

• 
  

  Application scenarios:

   Large-scale vector similarity search, often used in conjunction with other indexing structures such as IVF (called IVF_PQ), to compress vectors stored in inverted lists, or directly as an indexing method.

  
  

• 
  

  Application method:

   When creating an index in a vector database or library (such as Faiss), select the PQ or IVF_PQ type and specify parameters such as the number of subvectors  M  and the number of codewords  K per subspace (usually K=256, corresponding to 8 bits). Choosing appropriate parameters requires a trade-off between compression rate, memory usage, search speed, and accuracy.

  
  

---




6. Large Language Models (LLMs)


Core idea: Use ultra-large-scale neural networks (usually Transformer architecture) and massive text data for pre-training to learn the statistical laws, grammatical structure, semantic relations and even certain world knowledge and reasoning ability of the language, so that it can understand and generate natural language.




• 
  

  Large Language Model (LLM):

  
  

• 
  

  Content Generation:

   Write articles, emails, code, poetry, advertising copy, etc.

  
  

• 
  

  Question and answer and dialogue:  intelligent customer service, chatbots, virtual assistants.

  
  

• 
  

  Text Summarization and Translation.

  
  

• 
  

  Sentiment analysis and text classification.

  
  

• 
  

  Generator in RAG:  Generates answers based on the retrieved context.

  
  

• 
  

  Application principle:

   Self-supervised learning based on the Transformer architecture. The model learns language representation by predicting the next word in the text (Causal LM, such as GPT) or masked words (Masked LM, such as BERT) during the pre-training phase. The huge number of model parameters and training data enables it to capture complex language phenomena.

  
  

  
  

  
  

  
  

• 
  

  Application method:

   Call pre-trained models through APIs (such as OpenAI API, Google Gemini API), or download open source models (such as Llama, Mistral) for local deployment and use/fine-tuning. Prompt Engineering is required to guide the model to produce the desired output.

  
  

• 
  

  Finetuning:

  
  

• 
  

  Specific area questions and answers:

   Fine-tuned for specialized fields such as medicine, law, and finance.

  
  

• 
  

  Specific style or role-playing:  Have the model imitate the speaking style of a specific person.

  
  

• 
  

  Improving performance on specific tasks:  e.g. fine-tuning on specific types of summarization or translation tasks.

  
  

• 
  

  Instruction following:  Through instruction tuning, the model can better understand and execute user instructions.

  
  

• 
  

  Application principle:

   Based on the pre-trained general LLM, additional training is performed using labeled datasets for specific tasks or domains. This allows the model to adjust its parameters to better adapt to the characteristics of the target task and data distribution, while retaining the general language capabilities learned in the pre-training phase.

  
  

  
  

  
  

  
  

• 
  

  Application method:

   Prepare a labeled dataset (input-output pairs) for the target task, select a suitable pre-trained model, and fine-tune it using a specialized training framework (such as the  Trainer API  of the  Hugging Face transformers library  ). You need to adjust hyperparameters such as the learning rate and number of training rounds. The fine-tuned model can be deployed and used.

  
  

• 
  

  Multi-modal:

  
  

• 
  

  Application principle:

   Refers to the ability of an AI system to process and integrate inputs from multiple senses (text, images, audio, video, etc.). Multimodal LLMs typically have encoders that can process inputs from different modalities and fuse these representations into a common space, which is then processed and generated by the core language model.

  
  

• 
  

  Application scenarios:  visual question answering, image description, text-to-image generation, multimodal dialogue systems, multimodal RAG.

  
  

• 
  

  Application method:  Use LLM APIs that support multimodal input (such as GPT-4V, Gemini Pro Vision) or deploy corresponding open source models. When inputting, you need to provide data of different modalities (for example, text and image URLs or Base64 encoding) as required by the model.

  
  

• 
  

  Embedded model (LLM context):

  
  

• 
  

  Application principle:

   Inside LLM, the input text is first tokenized, and then each token is converted into a vector embedding. These initial embeddings (usually including word embeddings, position embeddings, paragraph embeddings, etc.) are the starting point for LLM to process sequence information, and subsequent Transformer layers will continuously transform and enrich these embedding representations. At the same time, independent embedding models (such as Sentence-BERT) are often used in the retrieval stage of RAG to specifically generate text representations for similarity search.

  
  

• 
  

  Application scenarios:  An essential part of LLM internal processing flow; the retrieval component of the RAG system.

  
  

• 
  

  Application method:  For LLM internal embedding, it is an inherent part of the model. For RAG retrieval, a separate, high-quality text embedding model needs to be selected and used.

  
  

• 
  

  Block (LLM context):

  
  

• 
  

  Application principle:

   Since the Transformer architecture of LLM usually has a fixed context window size limit (such as 4k, 8k, 32k, 128k tokens), it cannot process very long documents at once. Chunking is the process of splitting long documents into segments smaller than the context window limit so that the most relevant chunks can be processed chunk by chunk or retrieved in RAG.

  
  

• 
  

  Application scenarios:  Processing documents that exceed the LLM context length; preparing knowledge base data for the RAG system.

  
  

• 
  

  Application method:  Choose a suitable block strategy (fixed size, recursive, semantic, etc.), and use the block tools provided by frameworks such as LangChain, LlamaIndex, etc. Parameters such as block size and overlap need to be considered to balance information integrity and processing efficiency.

  
  

• 
  

  Generative AI:

  
  

• 
  

  Application principle:

   Different from discriminative AI (such as classification and regression), the goal of generative AI is to learn the underlying distribution of training data and to sample and generate new data that is similar to but different from the training data. LLM generates text by learning the probability distribution of language, and diffusion models generate images by gradually removing noise.

  
  

• 
  

  Application scenarios:  content creation, art design, data enhancement, virtual world construction, drug discovery, etc. LLM is an important branch.

  
  

• 
  

  Application method:  Use the corresponding generation model (LLM, Stable Diffusion, Midjourney, etc.) to guide the generation process by providing prompts or conditions.

  
  

• 
  

  Transformer Model:

  
  

• 
  

  Application principle:

   At its core is the self-attention mechanism, which allows the model to dynamically calculate the importance (attention weight) of all other elements in the sequence to the current element when processing each element in the sequence, and aggregate information accordingly. This enables the model to effectively capture long-distance dependencies and is easy to calculate in parallel. Multi-Head Attention allows the model to focus on information from different representation subspaces. Combined with Positional Encoding to process sequence order. Encoder-Decoder structure or only encoder/decoder structure is widely used.

  
  

• 
  

  Application scenarios:  The infrastructure of almost all modern top LLMs; machine translation, text summarization, question-answering systems, and many other NLP tasks.

  
  

• 
  

  Application method:  It is integrated into various LLM and NLP models as the underlying architecture. Researchers and developers usually use pre-trained models based on Transformer and adapt them to specific tasks through fine-tuning.

  
  

---




7. Information Retrieval/Search


Core idea: Find relevant and useful information (usually documents or data fragments) from large-scale data sets based on the user's information needs (usually expressed as queries).




• 
  

  Reranking:

  
  

• 
  

  Application principle:

   The idea of ​​phased retrieval. The first phase (recall/retrieval) uses fast but relatively rough methods (such as BM25, ANN vector search) to filter out candidate sets (such as Top K=100) from massive data. The second phase (reranking) applies a more complex, more accurate but computationally more expensive model (such as the Transformer-based cross encoder) to re-score and rank this smaller candidate set to obtain the final result (such as Top N=10). The cross encoder inputs the query and each candidate document into the model at the same time, which can make a deeper relevance judgment.

  
  

• 
  

  Application scenarios:  Improve the accuracy and relevance of the final results of search engines, recommendation systems, and RAG systems.

  
  

• 
  

  Application method:  After the initial retrieval, the query and candidate document pairs are fed into a re-ranking model (such as  the Cross-Encoder model in the sentence-transformers  library) to re-rank the candidate set according to the relevance scores output by the model.

  
  

• 
  

  Retrieval Enhanced Generation (RAG):

  
  

• 
  

  Enterprise Intelligence Q&A:

   Answer employee or customer questions based on your internal document repository.

  
  

• 
  

  Knowledge assistants in specific fields:  such as medical and legal consulting assistance.

  
  

• 
  

  Question-answering system that requires citing sources or up-to-date information:  Makes up for LLM knowledge cutoff and illusion questions.

  
  

• 
  

  Application principle:

   It combines the advantages of information retrieval (finding relevant information from a knowledge base) and natural language generation (generating fluent and coherent answers using LLM). Basic process: Receive user query -> Use the query (after embedding) to retrieve relevant document fragments (context) in the vector database (or other knowledge base) -> Combine the original query and the retrieved context into an extended prompt -> Input this prompt into LLM -> LLM generates the final answer based on the provided context.

  
  

  
  

• 
  

  Application method:

  
  

• 
  

  Building a knowledge base:

   Prepare documents, segment them, embed them, and store them in a vector database.

  
  

• 
  

  Implement the retriever:  Select the embedding model and vector database to implement the function of retrieving the Top K relevant blocks based on the query.

  
  

• 
  

  Implement the generator:  select LLM, design an appropriate prompt template, integrate the query and retrieved context, and input it into LLM.

  
  

• 
  

  Integration and Optimization:  Use frameworks such as LangChain, LlamaIndex to simplify the building process, and apply various advanced RAG techniques for optimization.

  
  

---










---




8. Embedding Types




Core idea: Develop vector embedding representation methods with different characteristics to meet different needs (semantic capture, keyword matching, storage efficiency, representation granularity).




• 
  

  Variable Dimensions:

  
  

• 
  

  Resource-constrained devices:

   Use low-dimensional prefixes for inference on edge devices or in memory-constrained environments.

  
  

• 
  

  Multi-stage search:

   First use low-dimensional embedding for fast rough screening, and then use high-dimensional embedding for precise sorting.

  
  

• 
  

  Adaptive Search:

   Select an appropriate dimension based on the complexity of the query or the accuracy requirement.

  
  

• 
  

  Application principle:

   Allows the dimension of the embedding vector to be dynamically adjusted as needed. For example, the embedding trained by Matryoshka Representation Learning (MRL) has a prefix of the vector (such as the first 64 dimensions, the first 128 dimensions) that is itself an effective, low-dimensional embedding. This provides a flexible trade-off between computational resources/latency and representation accuracy.

  
  

  
  

• 
  

  Application method:

   Use pre-trained models that support variable dimensions (such as some models trained based on MRL). When storing, you can store the complete embedding, and truncate prefixes of different lengths as needed during queries or downstream tasks.

  
  

• 
  

  Sparse Embeddings:

  
  

• 
  

  Searches that require an exact match of a term, ID, or code.

  
  

• 
  

  The keyword search part in hybrid search,

   Complementary to dense embeddings.

  
  

• 
  

  Fields such as law and medicine that require precise terminology retrieval.

  
  

• 
  

  Application principle:

   It mainly captures information at the vocabulary level. The vector dimension is equal to the vocabulary size. Only the dimension corresponding to the word that appears in the document has a non-zero value (usually the weight of the word, such as the BM25 score or the learned weight such as SPLADE). It is good at exact keyword matching.

  
  

  
  

• 
  

  Application method:  Use TF-IDF, BM25 algorithm to calculate and generate; or use specially trained sparse representation models (such as SPLADE) to generate learned sparse vectors. Storage and query usually require specific data structures (such as inverted indexes).

  
  

• 
  

  Quantized Embeddings:

  
  

• 
  

  Very large-scale vector datasets:

   When billions or more vectors need to be stored and retrieved, memory and storage costs are the main bottlenecks.

  
  

• 
  

  Scenarios that require low latency queries:

   Quantized distance calculations are usually faster.

  
  

• 
  

  Memory-constrained environments.

  
  

• 
  

  Application principle:

   By reducing the precision of the values ​​in the vector (such as float32 -> int8 or binary) or using a codebook (such as PQ), the storage size of the vector is greatly compressed and the distance calculation is accelerated (such as int8 calculation is faster, and binary Hamming distance calculation is extremely fast). This is a lossy compression, which requires a trade-off between compression rate and accuracy loss.

  
  

  
  

• 
  

  Application method:  When creating an index in a vector database (such as Milvus, Qdrant) or library (such as Faiss), select an index type that supports quantization (such as  IVF_PQ ,  ScalarQuantizer ,  IndexBinaryFlat ). You need to configure quantization parameters (such as M and K for PQ, or the target number of bits int8/binary). The quantization process is usually completed during the index building phase.

  
  

• 
  

  Multi-vector Embeddings:

  
  

• 
  

  Passage Retrieval:

   It often outperforms a single vector representation when exact matches to specific phrases or terms in a query are required.

  
  

• 
  

  Question answering systems that require fine-grained relevance judgment.

  
  

• 
  

  Search tasks that are sensitive to word order or exact wording.

  
  

• 
  

  Application principle:

   Unlike compressing the entire text (such as sentences and paragraphs) into a single vector, the multi-vector method generates separate vectors for smaller units in the text (such as each word or tag). The "late interaction" strategy is adopted during retrieval, that is, the vector of each word in the query is compared with the vector of each word in the document, and these fine-grained similarity scores are aggregated to more accurately capture word matching and contextual relationships.

  
  

  
  

  
  

  
  

• 
  

  Application method:  Use models that support multi-vector representation (such as ColBERT). Specialized index structure and query logic are required to support later interactive calculations. Compared with single-vector retrieval, the amount of calculation is usually greater.

  
  

• 
  

  Dense Embeddings:

  
  

• 
  

  Semantic search (core application):

   Understand user intent and find conceptually related content.

  
  

• 
  

  NLP tasks such as text clustering, classification, and sentiment analysis.

  
  

• 
  

  Content representation in recommender systems.

  
  

• 
  

  The semantic retrieval part of the RAG system.

  
  

• 
  

  Application principle:

   Captures the deep semantic meaning of text or data, rather than superficial vocabulary. The vector dimension is relatively low (hundreds to thousands of dimensions), and most elements are non-zero. It is learned by deep learning models on a large amount of data, and objects with similar meanings are close in distance in the vector space.

  
  

  
  

• 
  

  Application method:  Use pre-trained dense embedding models (such as Sentence-BERT, Universal Sentence Encoder, OpenAI Ada embeddings, etc.) to generate vectors, store them in a vector database, and query them using metrics such as cosine similarity.

  
  

• 
  

  Binary Embeddings:

  
  

• 
  

  Extremely resource-constrained environments:

   Such as mobile devices or embedded systems.

  
  

• 
  

  Very large datasets that are extremely sensitive to storage costs.

  
  

• 
  

  Scenarios where a certain loss of accuracy can be accepted in exchange for extremely high speed.

  
  

• 
  

  Application principle:

   The extreme form of quantization compresses each dimension of the vector to only 1 bit (0 or 1). It is very memory efficient, and the distance between vectors can be calculated using the very fast Hamming distance (calculating the number of different values ​​on corresponding bits) or Jaccard similarity.

  
  

  
  

• 
  

  Application method:

   Use a model or technique that specifically generates binary embeddings. Need to use an index that supports binary vectors and Hamming distance (such as Faiss's  IndexBinaryFlat  or LSH-based binary indexing). The precision loss is usually more noticeable than int8 or PQ quantization.

  
  

  
  

  
  

  
  

---




9. Chunking Techniques




The core idea is to split large documents or data streams into smaller, more easily processed units (chunks) to adapt to model processing limitations (such as context windows) and optimize retrieval efficiency and relevance.




• 
  

  Semantic Chunking:

  
  

• 
  

  Application principle:

   Try to divide the boundaries based on the semantic content of the text, with the goal of each chunk containing a relatively complete and coherent unit of thought or topic. This is usually achieved by analyzing the similarity of the embedding vectors between sentences or paragraphs, and splitting where the similarity drops significantly.

  
  

• 
  

  Application scenarios:

   When dealing with texts with strong narrative and close logical connections between paragraphs (such as articles and reports), we hope that the context retrieved by RAG is as semantically complete as possible.

  
  

• 
  

  Application method:

   Use NLP libraries (such as spaCy) to perform sentence segmentation, calculate the embedding similarity of adjacent sentences (or sentence groups), set a similarity threshold or detect mutation points to determine the chunk boundaries. Some frameworks (such as LlamaIndex) provide experimental semantic chunkers.

  
  

• 
  

  Recursive Chunking:

  
  

• 
  

  Application principle:

   Takes a set of delimiters arranged in priority order (e.g.  \n\n ,  \n ,   (space),  .,  etc.), and tries to split the text with the highest priority delimiter first. If the split block is still too large, recursively splits the block with the next priority delimiter until the block size meets the requirement or no lower priority delimiter is available. Try to respect the natural structure of the text as much as possible while controlling the size.

  
  

• 
  

  Application scenarios:

   It can process texts that have certain structures (such as paragraphs and line breaks) but are not completely regular. It can also provide a relatively robust way to segment texts when the specific structure of the text is unknown.

  
  

• 
  

  Application method:

   This is the default or recommended method commonly used in frameworks such as LangChain and LlamaIndex. You need to specify the chunk size ( chunk_size ), chunk overlap ( chunk_overlap ), and a list of delimiters.

  
  

• 
  

  LLM-based chunking:

  
  

• 
  

  Application principle:

   Use the LLM's ability to understand the text content to guide the chunking process. You can use carefully designed prompts to let the LLM directly divide the text into semantically coherent chunks, or identify logical chapters or major topics in the text as the basis for chunking.

  
  

• 
  

  Application scenarios:

   Processing documents with complex structures that require deep understanding to be segmented reasonably (such as legal contracts and research papers); scenarios that pursue the highest quality semantic blocks.

  
  

• 
  

  Application method:

   Design a suitable prompt and call the LLM API for chunking. This is usually slower and more costly than algorithm chunking, and the benefits need to be carefully evaluated.

  
  

• 
  

  Fixed size chunks:

  
  

• 
  

  Application principle:

   The simplest method is to split the text according to a fixed number of characters or tokens. Usually an overlap is set so that the end of each block is the same as the beginning of the next block to reduce the risk of information being hard cut off at the block boundary.

  
  

• 
  

  Application scenarios:

   It processes plain text with no structure or unreliable structure information; it is simple and fast to implement and serves as a baseline method.

  
  

• 
  

  Application method:

   Directly cut by length, set  chunk_size  and  chunk_overlap  parameters. Pay attention to the differences in characters and words in different languages.

  
  

• 
  

  Chunking based on document structure:

  
  

• 
  

  Application principle:

   Use the document's inherent structural markers (such as HTML tags, Markdown headings, JSON keys, PDF bookmarks, or paragraph formatting) as chunk boundaries.

  
  

• 
  

  Application scenarios:

   Process well-formatted and clearly structured documents, such as web pages (HTML), Markdown files, code files, and structured PDF reports. The logical hierarchy of the original text can be preserved to the greatest extent.

  
  

• 
  

  Application method:

   Use the corresponding document parsing library (such as BeautifulSoup for HTML, PyMuPDF for PDF) to extract the structural information and the corresponding content blocks. For example, you can divide the content into blocks according to HTML  tags such as <p> ,  <div> ,  <h1>  , etc. or Markdown heading levels.

  
  

---




10. Advanced RAG Techniques




Core idea: Optimize each link of the basic RAG process (query understanding, data preparation, retrieval, generation) to improve the accuracy, relevance, reliability and efficiency of the final answer.










• 
  

  Reasoning and Action (ReAct):

  
  

• 
  

  Application principle:

   LLM is not only passively generating answers based on the retrieved context, but also actively planning and executing a series of “thinking-acting-observing” steps to solve the problem. Actions can include calling external tools such as search engines, calculators, database query APIs, etc.

  
  

• 
  

  Application scenarios:

   Handle complex queries that require multi-step reasoning, require external real-time information, or require interaction with external systems to answer. For example, "Find the latest stock price of company X and compare it to company Y."

  
  

• 
  

  Application method:

   Use a framework that supports the ReAct mode (such as LangChain Agents) or implement the prompt loop yourself. You need to provide the LLM with the available tool set and its description, and design prompts that can guide it to perform ReAct reasoning.

  
  

• 
  

  Thinking Tree (ToT):

  
  

• 
  

  Application principle:

   Allow LLMs to explore multiple branches of possibility during reasoning. For each step, the LLM can generate multiple "ideas" (possible next steps or solutions), then evaluate these ideas (perhaps through self-reflection or calling an external evaluator), and choose the most promising branch to continue exploring, even backtracking.

  
  

• 
  

  Application scenarios:

   Solve complex problems or generative tasks that require exploratory thinking, do not have a single correct path, or require evaluating multiple solutions. In RAG, this can be used to explore different interpretations of a query or evaluate combinations of multiple retrieved context fragments.

  
  

• 
  

  Application method:

   Implement complex prompting strategies or use specialized agent frameworks to manage state, generate, and evaluate multiple concurrent thought paths. Computationally expensive.

  
  

• 
  

  Chain of Thought (CoT):

  
  

• 
  

  Application principle:

   Prompts such as “think step by step” guide the LLM to output detailed reasoning steps before giving the final answer. This explicit, decomposed thinking process helps the LLM understand the problem more accurately, use contextual information, and reduce reasoning errors.

  
  

• 
  

  Application scenarios:

   Improve LLM's performance on tasks that require logical reasoning, mathematical calculations, or complex instruction following; in RAG, help LLM better integrate and utilize retrieved contextual information to construct answers.

  
  

• 
  

  Application method:

   Include instructions to guide the LLM to think step by step in the final generation prompt. You can also use "Zero-shot CoT" (just add instructions) or "Few-shot CoT" (provide several examples with reasoning steps in the prompt).

  
  

• 
  

  Data cleaning:

  
  

• 
  

  Application principle:

   The quality of RAG's knowledge base directly affects the retrieval and generation results. "Garbage in, garbage out." Cleaning aims to remove noise, errors, and irrelevant information to improve data quality.

  
  

• 
  

  Application scenarios:

   The data preparation phase of the RAG system, especially when the source data come from multiple sources and have varying quality.

  
  

• 
  

  Application method:

   Use regular expressions, scripts or specialized libraries to remove HTML tags, advertisements, headers and footers, repeated spaces; perform spelling correction, standardize date formats, handle missing values, etc.

  
  

• 
  

  Data extraction and analysis:

  
  

• 
  

  Application principle:

   Accurately extract processable text content and metadata from various raw formats (PDF, DOCX, HTML, PPT, etc.) and possibly parse their structure.

  
  

• 
  

  Application scenarios:

   The first step in building the RAG knowledge base is to process source files that are not plain text.

  
  

• 
  

  Application method:

   Use Python libraries such as  PyPDF2 ,  python-docx ,  BeautifulSoup ,  unstructured.io  , etc. For text in scanned PDFs or images, you need to use OCR tools (such as Tesseract, PaddleOCR).

  
  

• 
  

  Data conversion:

  
  

• 
  

  Application principle:

   Convert the extracted and cleaned data into a unified and regular format (such as plain text, Markdown, structured JSON) to facilitate subsequent segmentation, embedding, and LLM processing.

  
  

• 
  

  Application scenarios:

   Ensure consistency of data format within the RAG knowledge base.

  
  

• 
  

  Application method:

   Write conversion scripts as needed, for example, converting extracted tables to Markdown format, or unifying all text into UTF-8 encoding.

  
  

• 
  

  Embedding model fine-tuning:

  
  

• 
  

  Application principle:

   A general-purpose embedding model may not be able to understand the subtle differences between terms or concepts in a specific domain well. By fine-tuning on (query, related documents) pairs in the target domain, the embedding model can better capture the semantic relationships in that domain, thereby improving retrieval accuracy.

  
  

• 
  

  Application scenarios:

   RAG systems in professional fields (such as medicine, law, finance, and scientific research) have extremely high requirements for retrieval relevance.

  
  

• 
  

  Application method:

   Prepare a dataset of domain-related queries and corresponding relevant/irrelevant documents, and   further train the pre-trained embedding model using libraries such as sentence-transformers (usually using contrastive loss, triple loss, etc.).

  
  

• 
  

  Distance Thresholding:

  
  

• 
  

  Application principle:

   The results returned by vector search are sorted by similarity (or distance). By setting a threshold and retaining only the results with a similarity higher than the threshold (or a distance lower than the threshold), you can filter out those "noise" results that are relatively close in the vector space but may not actually be relevant to the query.

  
  

• 
  

  Application scenarios:

   Improve the signal-to-noise ratio of the context fed into the LLM and avoid irrelevant information interfering with the generation.

  
  

• 
  

  Application method:

   After receiving the Top K results returned by the vector database, the similarity score of each result is checked, and only the results whose scores exceed the preset threshold are passed to the generation stage. The threshold needs to be adjusted based on experience or experiments.

  
  

• 
  

  Tips for Engineering:

  
  

• 
  

  Application principle:

   The output quality of LLM is highly dependent on the quality of the input prompts. In RAG, how to organize the original query, how to present the retrieved context, and what kind of instructions to give to guide LLM to generate answers based on this information are all part of prompt engineering.

  
  

• 
  

  Application scenarios:

   The generation stage of RAG is the key link in determining the quality of the final answer.

  
  

• 
  

  Application method:

   Design clear, specific prompt templates. Experiment with different ways of presenting context (e.g., numbered lists, separators), different wording of instructions (e.g., “Answer the question based on the following information,” “Summarize the following information”), whether to require citing sources, etc.

  
  

• 
  

  Context Compression:

  
  

• 
  

  summary:

   Each retrieved block is digested using another LLM.

  
  

• 
  

  Selective Extraction:

   Based on the overlap or relevance score with the query, only the key sentences in the chunk are extracted.

  
  

• 
  

  filter:

   Remove duplicate or highly similar blocks.

  
  

• 
  

  Use a specialized context compression tool such as LongLLMLingua.

  
  

• 
  

  Application principle:

   The retrieved context may be large, exceeding the LLM context window limit, or contain redundant information. Compression aims to reduce the length of the context while retaining the most important information to fit within the window limit, reduce cost, and improve the focus of the LLM.

  
  

• 
  

  Application scenarios:

   When the retrieved context is too long or contains much redundancy.

  
  

  
  

• 
  

  Metadata filtering:

  
  

• 
  

  Application principle:

   Leverage metadata associated with vectors stored in the vector database (such as document source, creation date, author, category tags, etc.) to filter before or while searching for vectors. This can greatly narrow the search scope and improve efficiency and relevance.

  
  

• 
  

  Application scenarios:

   When a user query implies a requirement for specific metadata (such as "find reports on Company A from the past week"); or when the RAG system needs to retrieve information from a specific subset (such as documents for a certain department).

  
  

• 
  

  Application method:

   When calling the vector database query interface, you can also pass in the metadata filtering conditions. Most modern vector databases support this feature.

  
  

• 
  

  Query route:

  
  

• 
  

  Application principle:

   For complex RAG systems with multiple knowledge base indexes (which may correspond to different domains, different data types, or different update frequencies), query routing aims to intelligently direct user queries to the most appropriate index or processing flow based on the intent or content of the query.

  
  

• 
  

  Application scenarios:

   Large-scale enterprise-level RAG systems; RAG applications that need to integrate different data sources (such as document libraries, databases, APIs).

  
  

• 
  

  Application method:

   Add a query classification step at the beginning of the RAG process (you can use simple rules, keyword matching, or LLM for intent recognition), and select the index and prompt template for subsequent use based on the classification results.

  
  

• 
  

  Data preprocessing:

  
  

• 
  

  Application principle:

   This is the foundational work for building a high-quality RAG knowledge base, covering all preparation steps from raw data to clean, well-organized text blocks ready for embedding.

  
  

• 
  

  Application scenarios:

   The initial phase of any RAG system build.

  
  

• 
  

  Application method:

   Comprehensively use data cleaning, extraction, analysis, conversion, segmentation and other technologies to form a complete data processing pipeline.

  
  

• 
  

  Autocut:

  
  

• 
  

  Application principle:

   (Speculatively) An automated method to further prune or filter out contextual information that is judged to be less relevant to the core intent of the query after retrieval and before generation, even if they are close in vector space. It may be based on more complex rules or models to judge the actual contribution of information.

  
  

• 
  

  Application scenarios:

   Further purify the context and reduce the burden of LLM in processing irrelevant information.

  
  

• 
  

  Application method:

   It may be a specific library, algorithm or service that implements this functionality. The exact method is unknown, but the goal is to optimize the signal-to-noise ratio of the context.

  
  

• 
  

  Query Rewrite:

  
  

• 
  

  Application principle:

   The user's original query may be vague, have typos, be too complex, or not in line with the expression habits of the knowledge base. Query rewriting uses LLM or other techniques to optimize, clarify, or decompose the original query to make it more suitable for subsequent retrieval (especially vector retrieval) and generation.

  
  

• 
  

  Application scenarios:

   Improved handling of unstructured, natural language queries; handling of complex or multi-intent queries.

  
  

• 
  

  Application method:

   Before searching, the user query is input into an LLM and it is prompted to rewrite, correct, clarify or decompose (such as HyDE - Hypothetical Document Embeddings, generating a hypothetical answer and then searching its embedding; or Step-back Prompting, generating a more general question before searching).

  
  

• 
  

  Query expansion:

  
  

• 
  

  Based on dictionary:

   Extended with synonym dictionaries such as WordNet.

  
  

• 
  

  Based on LLM:

   Let LLM generate words or phrases related to your query.

  
  

• 
  

  Embedded Space Extension:

   Find other word vectors near the query vector for expansion.

  
  

• 
  

  Add the expanded terms to the original query for retrieval (may need to adjust the weights).

  
  

• 
  

  Application principle:

   To improve recall, especially when the user query is short or does not contain all relevant terms, synonyms, related concepts, or terms can be automatically added to the original query.

  
  

• 
  

  Application scenarios:

   When there is concern that the initial query may not retrieve all relevant documents (recall first).