From traditional RAG to knowledge graph + agent, the success rate of knowledge base AI question answering has finally reached 95%, from the experience of Ant Group

introduction

This article is from the sharing of "Domain Knowledge Management and AI Question and Answer" by the knowledge base expert from Ant Group at the #QECon Shenzhen Station in late May   . It introduced the process from traditional RAG to knowledge graph to Search Agent, and gradually improved the accuracy of RAG from 60% to 95%. I learned a lot and highly recommend it!

Knowledge base question and answer business scenario

The AI ​​assistant shared by experts is based on the company's R&D knowledge base. Its main goal is to reduce the manpower costs of R&D consulting work orders, with tens of thousands of work orders per month.

Traditional RAG Optimization

To do knowledge base question and answer, RAG is indispensable. Like everyone else, everything started with the traditional RAG solution.

Traditional RAGs include:

• Offline document processing : document parsing, long document segmentation, multi-level summarization of text/paragraph/block, embedding storage
• Online retrieval generation : pre-intention recognition/question rewriting/tool ​​calling, multi-way retrieval and Rerank sorting, Qwen14B answer generation

After optimization, the success rate of the traditional RAG solution is:

• Document recall rate 80%+ 
• The final generation accuracy is 60%+ 

Since I also do RAG Q&A within the company, after the same little bit of optimization and repeated evaluation, the final success rate is similar to what he shared, so I was deeply touched to see this number, and it also increased my confidence in his experience.

Traditional RAG is not enough

Why is traditional RAG not enough?

Because traditional RAG cannot solve complex problem scenarios:

• Traditional RAG has a low success rate in cross-document recall
• User needs are becoming increasingly complex and require multiple documents or even tool calls to answer
• Knowledge assets are not organized, retrieval is inefficient, and it is difficult to realize their due value

Solution: Knowledge Graph + DeepSearch

1. Use the lightweight LightRAG solution to build a knowledge graph and solve the problems of semantic understanding alignment and knowledge cross-document  
2. Use DeepSearch iterative search solution, integrate multi-source and multi-round search results, and use the reasoning ability of large models to improve the accuracy of question answering  

Knowledge Graph

Implemented on the construction of knowledge graph:

1. Dynamic entity extraction :  Integrate domain terminology library with LLM to implement hybrid entity recognition technology to ensure the real-time and accuracy of knowledge graph.
2. Automatic extraction of relationship tags : optimize tag weights based on user behavior feedback and support incremental updates.

Implementation of knowledge graph-based retrieval:

1. Local retrieval :  quickly locate related entity subgraphs to improve recall rate.
2. Global retrieval :  Use semantic expansion driven by relationship tags to solve cross-document association problems.

Finally, the recall rate reached  95%+ by combining local, global and traditional RAG optimization solutions.

DeepSearch Agent

Optimization 1: Retrieval Agent Combined with Iterative Search Framework

In the DeepSearch solution, traditional RAG search (including sparse search and dense search), local graph search, global graph search, code search, etc. are all used as a search tool and given to the big model to select.

Based on reasoning capabilities, the large model combines the results of each round of retrieval to determine whether and what tools to use for the next round of retrieval.

Optimization 2: Domain assistant agent combined with deep customization tools

• Query understanding :  Let the model decide autonomously which tools to call to solve the problem.
• Contextual rewriting :  Use contextual information to rewrite queries, improve the match between queries and knowledge tags, and enhance understanding capabilities.
• Tool calling :  Introduce a tool calling mechanism to support more complex operation processes and improve problem-solving capabilities.
• Optimize matching :  Use domain graphs for rewriting; effectively solve the matching problem between colloquial expressions and professional documents, and significantly improve matching efficiency and accuracy.

Business implementation effect

The rate of solving complex problems has been significantly improved, the average response time has been greatly shortened, and the number of manual work orders has been reduced by 10%.

The business implementation covers various technology stack platforms including front-end and back-end, proving the generalizability of the solution.

My thoughts

As I have been following the development of RAG technology, I remember that there were cases where RAG technology and experience were publicly shared and achieved very high accuracy. The only case was the one shared by Linkedin, which achieved a recall rate of 85%+ based on Knowledge Graph. Later, Microsoft publicly released the popular GraphRAG solution.

This time, the sharing from the Ant expert is another real case that has achieved a 95% accuracy rate, also based on GraphRAG.

Although the GraphRAG solution is complex and the token cost is high, it is still worth a try because of its high accuracy.

Finally, even a powerful tool like GraphRAG is still a search tool. To solve the complex demands of real users, an agent with reasoning capabilities and tool calling capabilities is the ultimate answer.