2025 AI Agent Technology Stack Panorama

Understanding the AI ​​Agent Landscape

Although we see many agent technology stacks and market distribution maps, we tend to disagree with their classification methods and find that these classifications rarely truly reflect the tools that developers actually use. In the past few months, the AI ​​agent software ecosystem has made significant progress in memory capabilities, tool invocation, secure execution and deployment. Based on our more than one year of practical experience in the field of open source AI and more than seven years of AI research accumulation, we decided to share our own "agent technology stack" to present a more comprehensive technical panorama that is more in line with industry practices.

Evolution from Large Language Models to Intelligent Agents

Between 2022 and 2023, we witnessed the rise of LLM frameworks and SDKs such as LangChain (released in October 2022) and LlamaIndex (released in November 2022). At the same time, standardized platforms for calling LLM services through APIs have gradually matured, and technologies for autonomously deploying LLM reasoning (such as vLLM and Ollama) have also formed a stable ecosystem.

Entering 2024, the industry's focus has significantly shifted to AI "agents" and more general composite systems. Although the concept of "agent" has existed in the field of reinforcement learning for decades, in the era of ChatGPT, it has been redefined as a system driven by LLM that can autonomously output action instructions (tool calls). This paradigm that combines tool calls, autonomous operation, and memory capabilities marks the leap from basic LLM to agents, and has also spawned the rise of a new generation of agent technology stacks.

What is unique about the agent stack? Agents are a significantly more complex engineering challenge than basic LLM chatbots because they require state management (keeping message/event history, storing long-term memory, executing multiple LLM calls in the agent loop) and tool execution (safely executing actions output by the LLM and returning results). Therefore, the AI ​​agent stack is very different from the standard LLM stack. Let's start by disassembling today's AI agent stack from the bottom layer, the model serving layer:

Model Serving

At the core of an AI agent is a Large Language Model (LLM). To use an LLM, you need to deploy the model through an inference engine, most commonly as a paid API service.

• Closed-source API model inference service providers : OpenAI and Anthropic are in the lead with their private cutting-edge models (such as GPT-4 and Claude 3).

• Open source model API service providers : Platforms such as Together.AI, Fireworks, and Groq provide hosting services for open source weight models (such as Llama 3) through paid APIs.

• Local model inference engine :

      Hobbyists (“AI enthusiasts”) : Ollama and LM Studio are two popular tools that support running models locally on a personal computer (such as an M-series Apple MacBook).

      Production-level GPU deployment : vLLM is the mainstream choice for GPU-based deployment in production environments, while SGLang is an emerging project targeting similar developer groups.

      Runs on local personal devices .

Storage Layer

Storage is a fundamental building block of stateful agents - a core feature of an agent is its persistent state, including conversation history, memory, and external data sources for RAGs.

• Vector databases : Vector databases such as Chroma, Weaviate, Pinecone, Qdrant, and Milvus are widely used to store the agent's "external memory", enabling the agent to utilize data sources and dialogue history that far exceed the capacity limit of the context window.

• Vector extensions of traditional databases : Postgres (a relational database born in the 1980s) supports vector search through the pgvector extension. Postgres-based companies such as Neon (serverless Postgres) and Supabase also provide embedded vector search and storage services for intelligent agents.

Tools and libraries layer

The core difference between a standard AI chatbot and an AI agent is that an agent has the ability to call "tools" (or "functions"). In most cases, the mechanism for this operation is that the LLM generates a structured output (such as a JSON object) that specifies the function to be called and its parameters. A common misunderstanding about agent tool execution is that tool execution is not done by the LLM provider - the LLM is only responsible for selecting the tool to call and providing parameters. Agent services that support arbitrary tools or arbitrary parameters must use a sandbox (such as Modal, E2B) to ensure safe execution.

All agents call tools through the JSON Schema defined by OpenAI - this means that agents and tools from different frameworks are actually compatible with each other. For example, Letta's agents can call tools from LangChain, CrewAI, and Composio because they all follow the same Schema specification. As a result, the supplier ecosystem for common tools is growing rapidly:

• General tool library : Composio is a popular general tool library and also provides authorization management functions.

• Vertical Special Tools :

• Browserbase (web browsing tool)

• Exa (Web search tool)

As more agents are developed, we expect the tool ecosystem to continue to expand and provide new capabilities such as authentication and access control for agents.

Agent Framework

The agent framework is responsible for orchestrating LLM calls and managing agent states. Different frameworks have design differences in the following aspects:

1. Agent state management

   State serialization : Most frameworks introduce the concept of state "serialization" (such as saving the state as JSON or byte stream), allowing the agent's dialogue history, memory, and execution phases to be restored by loading the serialized file.

   Database persistence : Frameworks such as Letta store all states in the database (message table, agent state table, memory block table) without explicit serialization. This design supports direct query of states (such as retrieving historical messages by date) and affects the scalability of the system (handling long dialogue history or multi-agent scenarios) and the flexibility of state modification.

2. Context Window Structure

Each time the LLM is called, the framework "compiles" the agent state into the context window. Different frameworks organize the data in the context window (such as instructions, message buffers) in different ways, which directly affects the agent performance. It is recommended to choose a framework that can transparently manage the context window so that you can precisely control the agent behavior.

3. Multi-agent communication

• Message Queue : Llama Index implements agent communication through message queue.

• Explicit abstraction layer : CrewAI and AutoGen provide specialized multi-agent abstraction interfaces.

• Direct calling mechanism : Letta and LangGraph support agents to call each other directly, allowing centralized (through supervisory agents) or distributed communication.

• Compatibility trend : Most frameworks now support both single-agent and multi-agent scenarios, because a well-designed single-agent system should be able to be easily extended to a collaborative system.

4. Memory management methods

To break through the LLM context window limitation, different frameworks use different memory management techniques:

• RAG-based Memory : CrewAI and AutoGen rely entirely on retrieval-augmented generation techniques.

• Advanced memory technology : phidata and Letta integrate innovative methods such as self-editing memory (such as MemGPT) and recursive summarization.

• Automation Tools : Letta provides built-in memory management tools, supporting searching historical messages by text/data, writing to memory, and editing context windows.

5. Open source model support

• Implicit optimization by model providers : Mainstream model providers use techniques such as resampling output and prompt word engineering (such as "please output JSON") to ensure that tool calls are in the correct format.

• Framework adaptation challenges : Supporting open source models requires the framework to handle the above issues on its own, so some frameworks are limited to supporting major model providers.

Key considerations for choosing a framework

When building an agent today, framework selection should be based on specific needs:

• Application type : Conversational agent vs workflow automation

• Operating environment : Notebook experiments vs production-level services

• Model support : the need for open source weight models

The core differentiation of future frameworks will be in the deployment process, where design decisions around state/memory management and tool execution will be more decisive.

Agent Hosting and Services

Most current agent frameworks are still designed to run in a local environment, such as a Python script or Jupyter Notebook. However, we believe that agents in the future should be considered services that can be deployed to local or cloud infrastructure and accessed through REST APIs. Just as OpenAI's ChatCompletion API has become the industry standard for LLM services, we expect that a unified agent API standard will emerge in the future - although there is no clear leader in this field yet.

Core Challenges of Deploying Agent Services

Compared with deploying LLM services, the deployment complexity of agent services is significantly increased, mainly due to:

State Management :

• Applications may need to run millions of agents, each with a continuously growing conversation history, memory, and execution state.

• When moving from a prototype to a production environment, the state of the agent needs to be normalized (such as structured storage and index optimization) rather than simply stored temporarily.

Tools enforce safety :

• Tool dependencies (such as Python package versions and system environments) need to be explicitly stored in the database so that the service can rebuild the runtime environment.

• The execution environment needs to be isolated (for example, through Docker containers or security sandboxes) to prevent malicious code from penetrating.

API Standardization :

• Agent interactions must be implemented through strictly defined REST API interfaces rather than direct calls within scripts.

• It is necessary to support production-level API features such as asynchronous communication, timeout retry, and rate limiting.

Current Practice and Future Trends

• Current situation : Developers usually combine technology stacks such as FastAPI (building API layer), Postgres (state storage), Modal/E2B (secure execution) on their own, but this process is repetitive and error-prone.

• Framework evolution direction :

• Built-in production capabilities : Mainstream frameworks (such as LangChain, CrewAI) are gradually integrating database connectors, API generators, and deployment tools. For example, LangChain recently launchedlangserveThe module automatically converts the agent to a REST API.

• Abstraction of state management : The framework may introduce declarative state definitions (similar to Django models) and automatically handle serialization, version migration, and query optimization.

• Hybrid deployment mode : supports seamless switching between local (debug mode) and cloud (production mode), similar to PyTorchtrain/evalMode switch.

Key decision points

When choosing an agent hosting solution, you need to evaluate:

• State storage costs : Cost trade-offs between vector databases (such as Pinecone) and relational databases (such as Postgres)

• Execution environment isolation : the balance between security and overhead of lightweight sandboxes (such as E2B) and full containerization (such as Kubernetes)

• API governance requirements : Do you need enterprise-level features such as integrated authentication (OAuth), audit logs, and SLA monitoring?

In the future, the competitive focus of intelligent agent frameworks will shift from "prototype building capabilities" to "production readiness", and the maturity of deployment workflows will become a core differentiating factor.