Dify Basics | In-depth interpretation of RAG: Why is hybrid retrieval necessary?

In the RAG (Retrieval-Augmented Generation) architecture, one of the core methods of the retrieval phase is vector retrieval. The technical principle of vector retrieval is to divide the documents in the external knowledge base into semantically complete paragraphs or sentences, and convert these paragraphs into multi-dimensional vectors that can be understood by computers through embedding, and then perform the same conversion process on the questions raised by users. Then, by calculating the semantic similarity, the system can efficiently match the most relevant text paragraphs.

This vector-based retrieval method is particularly good at handling complex semantic relationships. For example, if a user queries "cat chasing mouse", the system can identify sentences that are semantically related to it, such as "cat hunting mouse", and the semantic similarity between them is high. Compared with "cat chasing mouse", sentences like "I like to eat ham" are much less relevant. After retrieving the most relevant text content, the RAG system will provide this information as context to the large model to help the model make more accurate answers.

Vector retrieval technology has shown significant advantages in many application scenarios, such as:

• Similar semantic understanding
  It can handle words with similar semantics such as "rat" and "mousetrap", "cheese", "Google" and "Bing", and can even understand the relationship between some synonyms.
• Cross-language understanding
  Supports multi-language retrieval, for example, you can obtain relevant content in English documents through Chinese queries.
• Multimodal Understanding
  It not only supports text, but also can realize similarity matching of multimodal information such as images, audio, and video.
• Fault Tolerance
  Able to handle spelling errors or ambiguous descriptions, reducing retrieval failures due to unclear expressions.

Although vector search performs well in many scenarios, it also has its limitations, especially when faced with some specific search requirements, the effect is not satisfactory. For example:

• Search for proper nouns or product names
  (e.g. “Elon Musk”, “iPhone 15”).
• Search for an abbreviation or phrase
  (such as "RAG", "RLHF").
• Search ID or version number
  (such as "gpt-3.5-turbo", "titan-xlarge-v1.01").

These scenarios are exactly what traditional keyword search is good at. Traditional keyword search technology has the following advantages:

• Exact Match
  When searching for names of people, product names, codes, etc., keyword searches can provide accurate matches.
• Handling matching of small numbers of characters
  When the query entered by the user consists of only a few keywords, vector search often has difficulty finding documents with strong relevance, while traditional keyword search can easily cope with it.
• Matching low frequency words
  Low-frequency words usually carry deeper semantic information. For example, in the sentence "Do you want to go for coffee with me?", the meanings of "drink" and "coffee" are obviously more important than "you", "want" and "do", and keyword search is good at capturing these key low-frequency words.

It can be seen that although vector search and keyword search each have their own areas of expertise, relying solely on one of the technologies will lead to unsatisfactory performance of the system in some scenarios. In order to give full play to the advantages of these two methods and make up for their shortcomings, hybrid search came into being.

Concept and Application of Hybrid Retrieval

Hybrid retrieval refers to combining different retrieval technologies to give full play to the advantages of each technology in the same retrieval task. For example, in the RAG system, we can combine vector retrieval with keyword retrieval, so that the system can take into account the advantages of both semantic matching and exact matching when searching. This approach can achieve the best results in a wider range of retrieval scenarios.

Specifically, the implementation of hybrid retrieval usually requires the establishment of two types of indexes in the database: vector index and keyword index . When the user enters a query, the system retrieves the most relevant text through these two retrieval methods. Then, the system will integrate the two search results and output the answer that best meets the user's needs. This combination not only ensures the high accuracy of the retrieved content, but also ensures semantic relevance.

Why do we need hybrid search?

Although vector search and keyword search have their own strengths, they cannot perfectly meet all search needs. Hybrid search combines the two methods to make up for the shortcomings of the two methods and give full play to their respective advantages. Specifically:

1. Make up for the lack of exact matching : For keywords such as proper nouns, abbreviations, IDs, etc., vector search may fail because it cannot perform exact matching. By introducing keyword search, the system can ensure that these exact matching search requirements are accurately responded to.

2. Improve query accuracy : For some queries containing low-frequency words or long-tail keywords, vector search can effectively understand the semantic similarity, but the accuracy may not be as good as keyword search. By combining the two technologies, hybrid search can ensure high-precision semantic understanding and retain the powerful ability of keyword matching.

3. Covering more complex scenarios : In some scenarios, relying solely on a single retrieval method may not be able to cope with complex query requirements. By introducing multiple retrieval algorithms, hybrid retrieval can better adapt to a variety of retrieval scenarios. For example, if a query involves complex entity relationships, it can be combined with knowledge graph and vector retrieval to further improve the retrieval effect.

Implementation and Challenges of Hybrid Retrieval

Implementing hybrid retrieval is not without its challenges. First, the system needs to maintain two indexes (vector index and keyword index) in the database at the same time, which increases storage and computing costs. Second, how to effectively integrate the two search results to ensure that the final output meets user needs also requires a carefully designed sorting and weighting mechanism.

In addition, the effect of hybrid retrieval not only depends on the advantages and disadvantages of the two retrieval technologies, but is also closely related to factors such as the quality of the data, the complexity of the retrieval task, and the user's query method. Therefore, when designing and implementing a hybrid retrieval system, multiple factors need to be considered to ensure its performance in practical applications.

Last words

In general, hybrid retrieval combines the advantages of vector retrieval and keyword retrieval to make up for their respective shortcomings, thereby improving the performance of large models in complex retrieval tasks. In practical applications, hybrid retrieval can not only improve retrieval accuracy, but also improve the flexibility and adaptability of the system, especially when facing diversified and complex query tasks, showing its huge advantages. With the continuous advancement of artificial intelligence technology, hybrid retrieval will play an important role in more fields and become one of the key technologies to promote the development of intelligent search and question-answering systems.

There is actually no clear definition for "hybrid retrieval". This article uses the combination of vector retrieval and keyword retrieval as an example. If other search algorithms are used in combination, it can also be called "hybrid retrieval". For example, the knowledge graph technology used to retrieve entity relationships can be combined with the vector retrieval technology.