Capability Grading of Large Model Applications

Grading the capabilities of large model applications is like grading students, which can help us better understand how capable they are. Capability grading can help us set goals, know what AI can do now, and what it needs to learn in the future. With a unified grading method, everyone can fairly compare the levels of different AIs and promote technological progress. At the same time, AIs of different grades are suitable for different tasks, which can help us find the most suitable helpers. In addition, capability grading makes it easier for ordinary people to understand the capabilities of AI and avoid excessive expectations or worries.

There are two common modes for the application of large models: RAG and Agent. The choice of RAG architecture depends on the specific problem to be solved and ensure that it is suitable for the task requirements. Nowadays, RAG with agent functions is becoming more and more important, which is very similar to the concept of "agent x". This "x" is like a universal toolbox that can be flexibly adjusted according to different scenarios to help us automatically complete tasks and make wise decisions, thereby improving efficiency. In addition, in order to deal with complex multi-part problems, it is critical to integrate document information from different sources. Simply put, these technologies are designed to make artificial intelligence smarter and more flexible to help us solve problems.

1. RAG Review

There are several key challenges in implementing an efficient RAG (retrieval-augmented generation) system: first, the system needs to be able to accurately find information relevant to the user's question; second, it must correctly understand the user's true intention; and finally, it must be able to leverage the reasoning capabilities of large language models (LLMs) to handle complex tasks. In order to improve reasoning capabilities, a method called "Agentic" can be used, such as ReAct, which solves problems by building a series of logical reasoning and operational steps. It should be noted that different LLM application scenarios may require different solutions, and no one method is suitable for all situations.

1.1 Context

Context refers to the relevant information accumulated during the conversation, which helps AI better understand the user's needs and make appropriate and coherent responses. This information includes what the user said before, the current task context, environmental factors, and other external data that may affect the conversation. By effectively processing context, AI can maintain the consistency and personalization of the conversation, adjust the answer according to the progress of the conversation, and make the entire communication process more natural and meaningful.

1.2 User Intent Detection

Often, poor system performance is due to missing the core of the user’s question or failing to accurately distinguish and apply multiple skills when faced with a task that requires a combination of skills. User intent refers to the real purpose or goal behind the user’s question, that is, what they hope to get or express through the question. Accurately identifying user intent is key to AI systems providing appropriate responses.

For more information about RAG, please refer to " Big Model Series - Interpretation of RAG ", " 10 Papers on RAG-2024Q1 ", " Chunking: Document Chunking in Big Model RAG System ", " Interpretation of GraphRAG " and " Applying Knowledge Graphs in Big Model RAG System ".

2. RAG capability classification

According to Microsoft's research results, RAG's capabilities can be divided into four levels based on the complexity of the search.

2.1 Level 1: Explicit Fact Query

This query is the simplest form, where the user directly asks for a specific fact that is clearly present in the data, without the need for additional reasoning. For example, a user asking "What is the diameter of the Earth?" simply requires finding the corresponding number in the data. The task of the RAG system is to locate and extract this directly present information, just like quickly finding a specific sentence in a book.

2.2 Level 2: Implicit Fact Query

This type of query is slightly more complex. The user's question does not directly point to a clear fact, but requires some background knowledge or logical reasoning to get the answer. For example, the user asks "What is the majority party in the country where Canberra is located?" To answer this question, the system needs to know that Canberra is the capital of Australia, and then infer the answer based on the current political party situation in Australia. This type of query may require extracting information from multiple places and performing simple logical connections.

At this level, the RAG system begins to show certain "intelligent agent" characteristics, because it not only needs to retrieve information, but also needs to perform some reasoning and logical judgment.

2.3 Level 3: Explainable Reasoning Queries

This type of query requires not only knowing the facts, but also understanding the logic and principles behind the facts and being able to provide a clear explanation. Answering such questions requires a combination of factual knowledge and domain-specific rules or guidelines, which are not usually present in ordinary language model pre-training.

For example, in a financial audit, a legal expert may need to determine whether a company's financial statements meet the standards based on compliance guidelines. This is not just a simple search for data, but also requires the application of professional rules to analyze and interpret.

Similarly, in a technical support scenario, the system may need to follow a troubleshooting process to help users solve problems, ensuring that each step complies with established operational specifications, thereby providing accurate and consistent responses. This type of query requires the system to not only have knowledge, but also the ability to apply this knowledge to solve practical problems.

2.4 Level 4: Implicit Reasoning Query

This type of query requires AI to not only see the surface information, but also dig deep into the underlying patterns and logic behind the data. It needs to infer complex principles that are not directly written out based on the context and observed patterns. These hidden principles often involve deep reasoning and logical connections that are difficult to find or extract directly.

For example, in IT operations, AI can summarize effective strategies by analyzing successful cases of solving similar problems in the past. It needs to discover patterns from a large amount of data rather than simply copying existing solutions.

In addition, in software development, AI can infer efficient problem-solving methods by studying past debugging cases. By integrating these implicit insights, AI can provide more refined and practical suggestions to help make smarter decisions. This type of inquiry reflects AI's ability to learn from data and extract wisdom.

Explainable and hidden rationales focus on the ability of RAG systems to understand and apply the logic behind data. These higher-level tasks require a deeper thinking process, often combining expert knowledge or extracting valuable insights from large amounts of unstructured historical data.

As can be seen from the previous examples, there is a clear distinction between tasks that directly query explicit facts (such as checking visa eligibility, which requires reference to the consulate’s official guidelines, which belongs to L3) and tasks that rely on implicit reasoning (such as analyzing the economic impact of a company’s future development, which requires combining financial reports with economic trends, which belongs to L4).

In either case, external sources of data are crucial—whether it be official documents or expert analysis reports. In these scenarios, providing rationales not only makes the answer more accurate, but also puts the answer in context, allowing users to understand not only the "what" but also the "why." This ability makes the answers of the agentized RAG system more comprehensive and credible.

3. AI Agent

AI Agent is an intelligent automated system that can understand and respond to complex problems, solve multifaceted challenges, and complete tasks that require reasoning, adaptation, and decision-making. Unlike traditional automation tools (which often rely on fixed rules and preset scripts), AI Agent uses machine learning (ML) and natural language processing (NLP) technology to continuously learn and improve itself. This ability makes AI Agent very flexible, able to cope with dynamic and unpredictable environments, and quickly adjust strategies as new information emerges.

For example, if an AI agent is tasked with providing customer support, it can learn from past conversations, improve its responses, and automatically adapt to each customer’s unique needs. This ability to both learn autonomously and operate independently makes AI agents ideal for complex environments, especially those that require a high degree of adaptability and a deep understanding of context.

3.1 Main Features of AI Agents

AI Agent is a "good helper" for enterprises. It can simplify work, improve customer service level, and increase team efficiency. Its advantages are mainly reflected in the following aspects:

• Flexible and adaptable: AI Agent can adjust its strategies based on the latest data and easily cope with various complex and changing scenarios.
• Task splitting: Break down large tasks into small steps, solve them step by step, and continuously optimize until the best solution is found.
• Understanding context: AI agents can “understand” the context of a conversation or task and give accurate responses even when questions are complex or ambiguous.
• Human-machine collaboration: When encountering difficult problems or requiring high precision, AI Agent can ask human experts for help, combining the efficiency of AI with human wisdom.
• Tool integration: It can connect to various external tools, databases and systems to perform calculations or obtain real-time data, making it more powerful.

However, using AI Agents also requires careful planning, such as controlling response time, ensuring transparency, and ensuring data quality. Only in this way can it really work.

3.2 Evolution of AI Agents: From Simple Automation to Complex Autonomous Systems

The development of AI Agents is inseparable from the progress of machine learning and natural language processing technologies, and also benefits from the need to adapt to complex scenarios in the real world. Early automation tools, such as RPA (Robotic Process Automation) and chain systems, can handle structured task processes, but lack the flexibility to deal with unpredictable situations. With the emergence of AI Agents, we now have intelligent systems that can handle fuzzy inputs, perform multi-step reasoning, and make decisions based on dynamically changing contexts.

Traditional automation tools rely on pre-set task steps, each of which is strictly executed according to fixed rules. For example, RPA automates repetitive tasks by simulating human interaction with software (such as logging into the system and copying data from one application to another). However, the limitation of RPA lies in its rigidity - once the workflow or conditions change, it needs to be reprogrammed, which makes it incapable of dealing with dynamic environments.

Compared with traditional RPA and chain systems, AI Agent has completely different capabilities. Next, we will discuss the differences between them in detail from multiple dimensions:

For more information about Agent, please refer to " AI-driven Data Analysis: Data Agent ", " Agent Application in Prompt Engineering ", " Agent Application Development Based on Large Model (LLM) " and " When You Ask About Agent Mechanism? Do You Mean Agent, Proxy, Broker or Delegate? " and other articles.

4. Five levels of autonomy of AI Agents

AI agents can be divided into five levels of autonomy, each level representing the ability to act independently and handle complex tasks.

4.1 Level 1: Reactive Agents

Reactive agents are the most basic type of AI agent. They work in a simple way: respond to specific inputs according to the rule of "if X happens, then do Y". These agents have no memory and cannot understand context, so they can only handle very simple tasks. Although they perform well in answering some common questions, they are overwhelmed when faced with more complex or flexible requests.

Key Features:

• Rules-based operation: can only react according to preset rules.
• No memory: Unable to remember past interactions or learn new information.
• Best for: Simple customer service tasks or day-to-day inquiries.

For example: a simple customer service bot can answer common questions like "What are your business hours?" or "Where is my order?", but it will have a hard time giving a useful answer to a slightly more complex question like "Why is my order delayed?"

4.2 Level 2: Contextual Agents

Contextual agents are smarter than reactive agents because they understand basic contextual information. Unlike agents that simply respond, they can make more reasonable decisions by analyzing clues in the environment. Although they still rely on rules, they can adjust their responses based on the user's history, location, and other conditions.

Key Features:

• Can use limited contextual information to improve the accuracy of responses.
• Behavior can be adjusted according to changes in the environment.
• Suitable for scenarios that require combining simple context to improve service quality.

For example , a virtual assistant could recommend nearby store opening hours based on the user's location, or provide more personalized suggestions based on the user's past interactions. This kind of intelligence can make services more intimate and useful.

4.3 Level 3: Adaptive Agents

Adaptive agents are like learning assistants that use machine learning to learn from past interactions and continuously improve their performance. They can adjust their behavior based on user feedback, making them ideal for scenarios that require flexibility. These agents are often used in customer service and support work, where they analyze user feedback to improve service quality.

Key Features:

• With machine learning, continuous improvement is possible.
• Optimize responses by analyzing user feedback and behavior patterns.
• Ideal for adaptable, data-dependent tasks.

For example: For example, a customer service robot can better understand customer needs and provide more precise assistance by analyzing past conversations and user feedback.

4.4 Level 4: Goal-Driven Agents

Goal-driven agents are like "self-motivated assistants" that are designed to independently achieve specific goals and solve problems through strategic approaches. Unlike agents that can only perform simple tasks or adapt to the environment, they evaluate multiple strategies and choose the one that is most likely to achieve the goal. This makes them particularly suitable for handling complex tasks that require multi-step planning and execution.

Key Features:

• Works independently and can evaluate different approaches to achieve goals.
• You can prioritize tasks and flexibly adjust strategies based on the results.
• Ideal for complex tasks that require strategic planning and step-by-step execution.

For example , a sales assistant robot can proactively recommend products and even suggest matching items based on the customer’s shopping history, helping the customer complete a complete set of clothing combinations and achieve their shopping goals.

4.5 Level 5: Fully Autonomous Adaptive Agents

Fully autonomous adaptive agents are like "super-intelligent assistants" that can perform complex tasks almost independently and with little human intervention. They can understand chaotic data, respond flexibly to unexpected situations, and adjust strategies based on real-time feedback. Such agents are well suited for high-stakes, fast-paced environments because they can react quickly and accurately.

Key Features:

• Capable of self-learning and adjusting behavior in real time.
• Proactively take action based on user behavior and context.
• Works efficiently in highly variable environments with little to no human supervision.

For example , a medical AI agent can monitor patients’ health data in real time, identify potential health risks, and provide preventive care recommendations or further examination plans based on each patient’s medical history and risk factors.

AI agents represent a major leap forward in business technology, automating complex, high-value tasks that were previously impossible for machines to perform. As machine learning, natural language processing, and computing power continue to advance, AI agents will become more intelligent and autonomous, better able to understand context, learn new knowledge, and make informed decisions.

Companies that adopt AI Agents will gain significant benefits, such as improving work efficiency, reducing operating costs, and improving customer satisfaction. As AI Agents continue to enhance their capabilities, we can foresee that they will play an increasingly important role in strategic decision-making, customer interaction, and cross-industry process optimization, becoming the core force driving corporate development.

5. Summary

Grading the capabilities of large model applications will not only help promote technological development, but also better match actual application scenarios, while also making it easier for the public to understand their value.

According to Microsoft's research, RAG's capabilities can be divided into four levels according to the complexity of the search: explicit fact query, implicit fact query, explainable reasoning query and implicit reasoning query. Regardless of the level, external data sources play a key role.

According to the autonomy of AI Agent, it can be divided into five levels: reactive agent, contextual agent, adaptive agent, goal-driven agent and fully autonomous adaptive agent. The future AI will be an ecosystem composed of multiple interconnected and highly autonomous agents. These agents will support and enhance human capabilities and provide more personalized, efficient and flexible new solutions.