Josh Tobin, head of OpenAI Agent, on strategies for building effective AI agents

The AI ​​wave is getting stronger and stronger! Sequoia Capital USA directly raised the banner of a trillion-dollar market at the AI ​​Summit, and at the top of the AI ​​wave is AI Agents - those intelligent agents that are endowed with the ability to understand, plan, and even autonomously perform complex tasks - which are increasingly becoming the focus of the technological revolution.

They are no longer jargon used in BP storytelling, but real forces that are reshaping the way we interact with the digital world. As a leader in this field, OpenAI has invested a huge amount of resources in cutting-edge exploration and practice. This time, we dug up Josh Tobin, the head of its AI agent research, and in his recent in-depth sharing, he provided us with a valuable window to peek into the core strategies, key insights, and practical experiences that OpenAI  follows behind  the internal operations of building truly effective AI agents.

Bottlenecks of directly calling existing large model API methods

"We had to face a reality at the beginning," Josh Tobin pointed out when reviewing the early exploration of building AI agents. "The initial model of simply building a fixed workflow around the Large Language Model (LLM) API quickly exposed its limitations." He admitted that although this method can quickly build a compelling demonstration, its reliability is often vulnerable in real, dynamic and uncertain application scenarios.

Tobin attributes this to two core pain points: "The first is the ' error accumulation ' effect. LLM may work well when handling single, isolated tasks, but when the task chain is extended and requires multi-step collaboration, the slightest deviation at each step may be amplified like a snowball, eventually leading to the collapse of the entire process."

He then added: "Second, and more fundamentally, is the 'misalignment of training and tasks' . Historically, most LLMs were not born to play the role of 'intelligent agents' with a high degree of autonomy. They were not designed to operate continuously in complex environments, learn lessons from inevitable mistakes, and recover flexibly. The lack of this core capability is a gap that traditional methods cannot overcome."

OpenAI’s core engine: end-to-end training and reinforcement learning for “agent capabilities”

What to do? The large language model obtained through pre-training is like a brain in a vat. If you tinker with the model output according to the existing model, it will be a probability game. OpenAI went straight to the core of the problem and completely revolutionized the model training paradigm. "We realized that the model must learn to 'do things', not just 'talk'," Tobin emphasized. As a result, the concept of end-to-end training came into being and quickly became a key strategy for OpenAI to tackle the capabilities of AI agents .

This means that instead of breaking down complex tasks into isolated sub-steps and training them separately, the agents are trained directly on the complete, complex workflows that they would need to complete in the real world. “In this way, the models are fully exposed to a variety of potential failure scenarios and edge cases during the ‘learning’ phase, which lays a crucial foundation for them to learn how to deal with unexpected situations and how to recover from errors, ” Tobin explained.

In the framework of end-to-end training, reinforcement learning plays the role of "soul injector". "This is not just about letting the model see more examples," Tobin pointed out, " More importantly, it is necessary to let the model learn in action and adjust its strategy according to the good or bad results ."

Through a carefully designed reward mechanism, AI agents are positively motivated during the training process for successfully completing tasks or achieving key phased goals. The magic of this mechanism is that it not only drives the model to actively explore and learn solutions that may be better and more efficient than the processes preset by human designers, but also gives the model a valuable "resilience" - the ability to recover from errors and adapt to unknown situations.

In another presentation at the Sequoia AI Summit by Dan, the head of RL research at OpenAI, he showed a graph showing that the duration of tasks that AI agents can complete is growing exponentially, doubling about every seven months. According to this graph, AI can currently handle tasks that last about an hour. What about next year? They can handle tasks that last about two to three hours.

He imagined trying to extrapolate this line. Just as Einstein needed eight years to think about relativity (OpenAI has a model called "Einstein v1907-Super H" that is doing this). From now on, to reach that level, we need about 16 doubling cycles. This means that in about nine years, we will have a model that can discover general relativity.

Tobin used OpenAI's "DeepResearch" agent as an example to vividly illustrate this point: "Imagine that if the results of the initial search are not ideal, an intelligent agent carefully tuned by reinforcement learning can rely on its accumulated 'experience' in training to keenly realize the shortcomings of the current strategy and actively adjust the search direction, try new keyword combinations, and finally dig out truly valuable information." He concluded: "This ability to 'learn from failure and evolve through practice' is the essence of the new generation of AI agents that distinguishes them from traditional models."

At the same time, OpenAI's new generation of basic models (such as the 03 model mentioned many times in the interview) have achieved a qualitative leap in the depth of understanding complex instructions, the accuracy of following multi-step processes, and the efficiency of recovering from errors. "They are trained to be able to 'see' signs of failure and quickly optimize subsequent behaviors accordingly, which was unimaginable before."

Train basic models with autonomous learning, reasoning, and trial-and-error capabilities

"Of course, all these advanced training strategies and exciting intelligent agent capabilities are inseparable from a solid foundation - that is the powerful basic model that we continue to invest in research and development," Josh Tobin did not shy away from the core position of the basic model (such as GPT-3, GPT-4 and its subsequent iterations). In his view, these basic models represent the highest level of current AI and have at least two core features that are critical to efficient AI agents:

One is their extraordinary generalization ability. "This means," Tobin explains, "that these large base models can show surprising adaptability and problem-solving potential even when faced with completely new tasks or highly customized scenarios that the developers did not explicitly anticipate during training. They are not just memorizing and reproducing, but also have a certain degree of 'learning from one example and applying it to other situations.'"

The second is excellent reasoning ability. "The internal logic of many intelligent tasks is extremely complex and the difficulty varies greatly," Tobin pointed out. "The model needs to have the ability of deep reasoning to navigate complex decision trees and accurately judge how much 'thinking resources' should be invested in each step of the process to ensure the smooth achievement of the final goal."

He added: "OpenAI's new models continue to improve in this regard. They are increasingly aware of 'when and what to think about'. This ability to dynamically adjust cognitive resources is crucial to improving the efficiency and success rate of intelligent agents."

Refine the capabilities of intelligent agents by iterating products in real scenarios

While OpenAI is strengthening the training of the base model, the product team is also looking for the best practice scenarios to give full play to the self-service capabilities of the model. They use a process that is closely integrated with actual product development and constantly evolves in the tempering of user feedback and real scenarios. "We firmly believe that only by applying theory to practice can we truly discover problems, verify ideas, and ultimately push the boundaries of technology," Tobin emphasized.

The series of intelligent products launched by OpenAI this year are a vivid manifestation of its core strategy. Many of these agents were proposed and supported by front-line employees, becoming a valuable "experimental field". Tobin introduced the three most important ones:

Deep Research

"This product has given us a full insight into the amazing power of reinforcement learning in processing complex information tasks," Tobin shared. It is not only a tool that goes beyond traditional search engines, but can also perform in-depth information mining and synthesis, and generate detailed and structured reports for users; it also demonstrates surprising "cross-border capabilities" in practical applications. "We found that,"

Tobin gave an example, "Users not only use it for market analysis and literature review, but also creatively apply it to assist coding - using it to search and understand complex code bases on GitHub, and even using it to mine those extremely rare and hidden 'cold knowledge' on the Internet."

In addition, the interactive design of "Deep Research" also reflects OpenAI's persistent pursuit of "learning and optimization": "By guiding the model and the user to conduct several rounds of questioning and clarification at the beginning of the task, we can help users express their real needs more accurately, thereby significantly improving the quality and relevance of the final research results."

Operator

"The development of Operator has taught us a lot about the extreme complexity of navigating a real, dynamic, and noisy network environment," Tobin said.

As an intelligent agent designed to perform various network operations (such as online restaurant reservations and price comparison shopping) on ​​behalf of users in a virtual browser, every successful interaction of Operator is the result of overcoming a series of technical challenges such as web page understanding, intent recognition, multi-step planning, and fault tolerance.

“Although Operator is still in the early stages of development,” Tobin admitted, “it has demonstrated that AI agents can be capable of performing some repetitive or unfamiliar network tasks, thereby creating real value for users, such as by allowing advanced users to provide customized instructions for specific websites.”

Codec CLI

"With Codec CLI, our original intention was to create a true programming partner that can work side by side with developers," Tobin described this open source local code execution agent.

He vividly likened it to an "intern with superhuman learning ability": "It can deeply understand and operate your local code base, and perform a series of tedious tasks for you, such as writing new features, applying code patches, and running unit tests, under the premise of obtaining user authorization." Its uniqueness lies in that "even when it comes to a brand new code base that has never been seen before (the so-called 'context-free' startup), Codec CLI can use its powerful learning and reasoning capabilities to quickly explore the file structure, understand the code logic, and independently complete coding tasks in a safe network sandbox environment, just like experienced developers, through standard command line tools."

Tobin further pointed out that the open source model of Codec CLI and its future development direction clearly indicate OpenAI’s vision: “We want to give models more persistent ‘memory’ capabilities, allowing them to learn and grow from continuous interaction with a specific code base;

At the same time, by providing more customized interfaces (such as APIs and MCPs) and actively embracing and absorbing the wisdom and strength of the open source community, we will jointly promote its evolution towards a smarter and more autonomous programming agent. "Its wide range of application scenarios has already begun to emerge,"

Whether it is a new project that needs to be started quickly from scratch, or dealing with unfamiliar code modules that are daunting to you, or automating repetitive coding tasks that engineers are generally reluctant to invest too much energy in (such as letting backend engineers handle some trivial adjustments on the front end), Codec CLI has shown great potential. "

Looking forward with cautious optimism: tool empowerment, trust building and paradigm innovation

When asked about the future of AI agents, Tobin’s response is full of considered optimism—an attitude that recognizes the challenges ahead but believes in the endless possibilities. “We are at an exciting inflection point,” he said, “but there are still many key problems that we need to overcome before AI agents can truly become a powerful tool for the general public.”

" Tool empowerment is the key to unleashing the potential of intelligent agents," Tobin emphasized. An effective AI agent requires not only a smart "brain" (i.e. a powerful general reasoning model), but also a set of flexible and powerful "hands" (i.e. the various tools required to complete real-world tasks). "The maturity and standardization of mechanisms such as the Model Control Protocol (MCP) are crucial to ensuring that models can call and use external tools safely, efficiently, and controllably. This is like equipping a smart apprentice with a sophisticated toolbox so that he can truly display his talents."

" Trust and security are the cornerstones of whether AI agents can be widely accepted ," Tobin said. How to enable humans to safely entrust potentially high-risk operations (such as authorization involving personal privacy and financial transactions) to AI agents is a major challenge facing the industry. "This not only requires us to establish clear and unambiguous guidelines, such as stipulating under what circumstances agents must obtain explicit permission from users to call certain sensitive tools; it also requires us to design reliable technical mechanisms to supervise and enforce these rules." He further pointed out that "the trust between users and AI agents is not achieved overnight. It needs to be gradually established and consolidated in continuous, transparent and predictable interactions. In the future, when the behavior of agents fails, how to clearly define the responsibilities of all parties (including users, model providers, tool providers, etc.) will also become an issue that requires the entire society to discuss in depth and jointly solve."

“ We have positive expectations about the balance between efficiency and cost ,” Tobin said. Although in the early stages, using advanced agent tools may bring certain learning costs and direct usage fees, “historical experience tells us,” he quoted, “as model capabilities continue to increase, algorithm efficiency continues to improve, and deployment costs are gradually optimized, the time saved, productivity improved, and new value created by AI agents for users will far exceed their direct economic costs. We are working hard to make this technology more and more accessible.”

"We are witnessing a profound change in developer workflows," Tobin said, expressing his high hopes for the new human-machine collaboration model represented by "Vibe Coding." " In the future, AI will increasingly take on those tedious and repetitive low-level code writing tasks , while the role and value of software developers will be more reflected in higher-level creative activities - such as system architecture design, precise definition of product functions, trade-offs between complex technical solutions, effective guidance and 'inspiration' of AI, providing high-quality feedback to optimize model behavior, and ultimately verifying the correctness and robustness of the entire system." He emphasized, "The importance of programming skills themselves will not disappear, because they are still the basis for understanding, debugging and mastering AI-generated code. But there is no doubt that the core competitiveness of developers will shift more to areas such as strategic thinking, problem definition and creative design."

"The impact of AI-driven educational change may be more far-reaching than we think," Tobin said. AI applications represented by tools such as ChatGPT and "Deep Study" are changing the traditional mode of knowledge acquisition and skill learning in an unprecedented way. "They can provide highly personalized, interactive learning experiences with unlimited questions and instant feedback, which are of immeasurable value in improving learning efficiency, stimulating learning interest, and even promoting educational equity."

"Ultimately," Tobin concluded, "through the organic combination of the above strategies and continuous iterative innovation, OpenAI is committed to transforming AI agents from an exciting cutting-edge scientific research concept, step by step, into a powerful tool that can create real value for thousands of industries and deeply empower humans. This road of exploration is undoubtedly full of unknowns and challenges, but the infinite possibilities it portends are enough to inspire us to move forward." In the view of Tobin and OpenAI, they have found a path of technological exploration leading to AGI, a path leading to a smarter, more efficient and more creative future.