10 thoughts from RAG founder on RAG Agent (Part 2)

6. Engineers spend a lot of time on “boring” things:

Engineers should focus on creating business value such as building processes, improving accuracy, and expanding applications, but in reality they often spend time on relatively basic and time-consuming tasks such as data chunking strategies, text cleaning, building connectors, configuring vector databases, adjusting prompts, and managing infrastructure. We should try to automate or platformize these tasks.

Practice Sharing

• There are many existing frameworks and tools that can help us automate these tasks. For example, we can use open source frameworks such as LangChain to handle data segmentation and text cleaning.
• Use tools such as Promptflow, LangSmith, and LangGraph Studio to track and debug large model responses.
• Frameworks like Promptflow can also be used to evaluate model performance.

Clever use of existing tools and frameworks can greatly reduce engineers' routine workload, allowing them to devote more time and energy to more valuable work.

7. Make AI Consumable:

Even if an AI system is deployed to production, it will not generate value if it is not easily used by users. Many times, the system usage rate is zero. The key is to integrate AI into users' existing workflows. Enterprise data + AI + integration = success.

Practice Sharing

Before finding an amazing AI implementation case, consider integrating AI into the existing enterprise workflow. Or just improve a small part of the enterprise, but make sure it can be used as much as possible. We once had a team develop an AI Sales bot, embedded in an existing corporate website, using the website's web content as a knowledge base to answer all questions from users browsing the website. At the same time, based on the conversation process with the user, it determines the timing to push information about related exhibitions. This facilitates the ultimate potential sales. This type of AI function is easily accepted by users and the effect can be seen in a short time.

8. Wow your users:

To make AI applications sticky, users need to experience the "wow" moment as soon as possible. For example, helping users find an important document that they didn't even know existed and had been buried for years and answering key questions. User experience design should focus on creating this early value.

Practice Sharing

The customer of the AI ​​Sales bot mentioned above, while demonstrating to their boss how to use sales tactics, was surprised to find a number listed in the bot's reply, saying that buying one of their products could save 20% of costs. What was the basis for this number? When the RAG-based bot returned the answer and the associated documents, they saw a document from many years ago. Such a good sales point was actually overlooked by them. This document was a customer feedback from many years ago, which mentioned the use case of this product and the cost saving figures.

9. Observability is more important than accuracy:

It is almost impossible to achieve 100% accuracy, but 90-95% may be possible. However, enterprises are more concerned about the impact of the inevitable 5-10% errors and how to deal with them. Therefore, observability, including understanding why the system gives a certain answer, providing traceability, and establishing audit trails, is more important than simply pursuing higher accuracy, especially in regulated industries.

Practice Sharing

• There is no need to pursue 100% accuracy. For AI systems based on LLM, users currently have a reasonable expectation of accuracy and will not apply such applications to businesses with high accuracy.
• Use tracking tools such as Promptflow or LangSmith to track the input and output of the model to help us understand the behavior of the model. The returned results must contain reference document information and give reasonable and well-founded answers so that users can trust the content generated by AI more.

10. Be Ambitious:

Many AI projects fail not because the goal is too high, but because the goal is too low. Don’t settle for solving simple problems like “who is my 401k provider?” Be brave enough to challenge the difficult problems that can bring real business transformation.

Practice Sharing

Give your AI a higher vision and goal, rather than just the few functions it implements.

• Based on the website AI Sales Bot, you can understand that its main function is almost the same as a question-answering robot. But if we define this as a digital employee, it can be the same as ordinary employees, with the goal of helping the sales team improve performance and helping customers better understand the product. Then we have more room for imagination to achieve such a goal. For example:
• Analyze the customer's role to proactively guide customer needs.
• Understand customer needs and proactively push relevant product information.
• Analyze user emotions based on conversation content and proactively push appropriate promotions.
• If you develop an AI private teacher for a school with the idea of ​​building an "educational equality" and build a professional teacher who can be online 24/7 and will never tire of explaining things to you, you will find that to achieve this goal, you will have to do much more than you think. For example:
• You need to consider how to enable the AI ​​teacher to understand students' emotions and how to enable him to have more natural conversations with students.
• You need to consider how to enable the AI ​​teacher to adjust teaching content based on students' learning progress and interests.
• You need to consider how to enable AI teachers to collaborate with other teachers, such as sharing teaching resources and exchanging teaching experiences.
• You need to consider how to enable AI teachers to communicate with parents, such as regularly sending students learning reports and providing learning suggestions.

Summary and Thoughts

According to a recent McKinsey study, AI enterprise applications can bring $4.4 trillion in global economic added value, but only one-quarter of enterprises benefit from it. This also reflects the huge uncertainty of AI applications at this stage. Although personal consumer applications can bring many interesting applications, and some companies have already made profits, enterprise applications are still in the exploratory stage due to their rigor and complexity.