The first "website" (WebAgent) designed specifically for AI was born: this may be a milestone

More and more intelligent agents are trying to obtain information directly from the Internet. Currently, there are many technologies available, such as Computer Use, Browser Use, etc. However, traditional websites are mainly designed for human users. If AI wants to use these websites, it often needs to simulate human browser behavior (for example, parsing HTML pages like a crawler), which is inefficient and complicated.

To solve this pain point, perhaps we need to build a WebAgent .

This article will introduce what WebAgent is and the technical details of the first WebAgent built on ANP, including how to discover and interact with WebAgent, the WebAgent authentication mechanism, and code examples for accessing related protocols. Finally, it will introduce how to build a data network designed specifically for AI.

What is WebAgent

The first time I came into contact with WebAgent was in the WebAgents community of W3C.

WebAgent is an intelligent agent built using existing Web infrastructure (such as PKI, DNS, HTTP, CDN, search engines, etc.). It runs on the Web and is designed to enable other intelligent agents to access and interact with it through the Web. It has both the openness of the Web and the AI ​​capabilities of intelligent agents . Unlike traditional websites designed with human users at the core, WebAgent is designed specifically for AI, providing public information that is readable and understandable to AI, as well as multiple types of interfaces including structured and natural languages, without the need for a user interface.

Compared with traditional websites, WebAgent has the following significant differences:

• Different object-oriented : Traditional websites mainly serve human users, who access and interact with them through browsers; WebAgent is designed specifically for AI, aiming to enable other intelligent agents to access and interact with it through the Web.
• Structured data format : The data returned by WebAgent is in a structured format (such as JSON, YAML, etc.), which is convenient for AI to read and process directly, while traditional websites usually return HTML, which requires additional parsing to extract the required information.
• Self-description capability : Each WebAgent provides a description file that details its functions, interface format, calling methods, etc., making it easier for other agents to understand how to interact with it.
• Identity verification mechanism : WebAgent has built-in authentication support for the access party to prevent information and interfaces from being abused. Existing websites often face the problem of being randomly crawled by web crawlers.

The emergence of WebAgent is of great significance to AI. In the past, AI needed to simulate human browsing to access web information, which was complicated and inefficient. WebAgent is a "native website" designed specifically for AI. AI can directly obtain data or perform tasks through standard interfaces, just like calling application services. This enables AI to use Internet information more efficiently and accurately, and developers can also optimize services for AI, no longer limited to human browsing patterns. WebAgent will become a basic component for building the Agentic Web, allowing AI to communicate and collaborate directly.

The first WebAgent was born (should be)

We developed the first ANP-based WebAgent, which is an intelligent website that provides weather information services.

As a demonstration, this WebAgent not only follows the native design of AI, but also adds a simple homepage so that human developers can intuitively understand its existence (although a homepage interface is not necessary for AI).

The following figure shows a schematic diagram of the WebAgent homepage, which you can also view by visiting the link https://agent-weather.xyz/:

(Figure: The first WebAgent - the demonstration interface of the weather agent. In actual interaction, AI will obtain weather data through the API interface it provides.)

This weather WebAgent is named  "Weather Agent" , which contains a virtual character "Xiaoqing" and is hosted on a dedicated domain name. The design of the weather agent fully complies with the ANP specification. Through this example, we will show how an ANP-based WebAgent runs, hoping to provide a reference for WebAgent developers and users.

It should be noted that, ideally, WebAgent does not need to have a human-accessible web interface; AI can communicate with it through the protocol. However, in order to facilitate debugging and let human developers understand how WebAgent works, the first WebAgent still provides an accessible homepage that displays its basic information and function introduction.

How to find a WebAgent

Since WebAgent is not centered on human visitors, how can we discover and locate these AI-oriented agent services?

To this end, ANP introduced a set of  intelligent agent discovery mechanisms , so that given a domain name, AI can find which WebAgents are under the domain name and their description files. In essence, ANP's intelligent agent discovery mechanism is also based on DNS, which is similar to the discovery mechanism of websites.

1. By domain name and .well-known Active Discovery:  Similar to some Internet services using .well-known Paths provide metadata (e.g. .well-known/robots.txt,.well-known/openid-configuration WebAgent also follows this idea. By convention, each domain name that hosts WebAgent can provide a list of agents :

https://<domain name>/.well-known/agent-descriptions

Accessing this path will return a discovery document in JSON-LD format , listing the addresses of all WebAgent description files published under this domain name. The details are defined in the ANP Agent Discovery Protocol specification (https://github.com/agent-network-protocol/AgentNetworkProtocol/blob/main/08-ANP-Agent-Discovery-Protocol-Specification.md) (Agent Discovery Protocol).

For example, for our weather agent domain agent-weather.xyz,access**https://agent-weather.xyz/.well-known/agent-descriptions**, will return something like the following:

{
  "@context" : {
    "@vocab" :  "https://schema.org/" ,
    "ad" :  "https://agent-network-protocol.com/ad#"
  },
"@type" :  "CollectionPage" ,
"url" :  "https://agent-weather.xyz/.well-known/agent-descriptions" ,
"items" : [
    {
      "@type" :  "ad:AgentDescription" ,
      "name" :  "Weather Agent" ,
      "@id" :  "https://agent-weather.xyz/ad.json"
    }
  ]
}

The above example shows that agent-weather.xyz Under this domain, there is an Agent named "Weather Agent", and its description file is located at https://agent-weather.xyz/ad.json(We usually also refer to this description file as  AD (Agent Description) ). If there are multiple agents, they will all be listed in items In this way, the AI ​​knows which URL to fetch next to learn more about the agent.

It is worth mentioning that the use of .well-known/agent-descriptions As an entry point, AI can find the list of WebAgents in a domain name without the need for humans to provide specific paths. This also makes it easier for search engines to index it.

2. Passive Discovery:  In addition to actively querying the .well-known path of a domain name, WebAgents can also submit their information to a dedicated agent search service or registry to be passively discovered. For example, a WebAgent can submit its description file URL to a public agent search engine so that other agents can search for it using keywords (such as "weather"). This is a type of passive discovery, which allows WebAgents to actively register themselves, making it easier for agents in the network to find each other.

Using the two discovery mechanisms designed by ANP, an agent can easily find the WebAgent it wants to access through the domain name or search engine.

How to interact with WebAgent

After finding the WebAgent, the next step is how to interact with it . At this time, you need to use the description file provided by the WebAgent (that is, the one found above). ad.json).

The agent description file details the capabilities and interface definitions of the WebAgent, which is equivalent to a manual . After reading this manual, the AI ​​can obtain the capabilities that the agent can provide according to the data and interfaces provided in it, and communicate with the agent.

We use the description file of the weather agent https://agent-weather.xyz/ad.json(https://agent-weather.xyz/ad.json) as an example:

{
  "@context" : {
    "@vocab" :  "https://schema.org/" ,
    "did" :  "https://w3id.org/did#" ,
    "ad" :  "https://agent-network-protocol.com/ad#"
  },
"@type" :  "ad:AgentDescription" ,
"@id" :  "https://agent-connect.ai/agents/travel/weather/ad.json" ,
"name" :  "Weather Agent" ,
"description" :  "Weather agent, providing weather information query services for cities across the country." ,
"version" :  "1.0.0" ,
"owner" : {
    "@type" :  "Organization" ,
    "name" :  "agent-connect.ai" ,
    "@id" :  "https://agent-connect.ai"
  },
"ad:securityDefinitions" : {
    "didwba_sc" : {
      "scheme" :  "didwba" ,
      "in" :  "header" ,
      "name" :  "Authorization"
    }
  },
"ad:security" :  "didwba_sc" ,
"ad:interfaces" : [
    {
      "@type" :  "ad:StructuredInterface" ,
      "protocol" :  "YAML" ,
      "url" :  "https://agent-connect.ai/agents/travel/weather/api_files/weather-info.yaml" ,
      "description" :  "YAML file of OpenAPI that provides weather query service."
    },
    {
      "@type" :  "ad:StructuredInterface" ,
      "protocol" :  "YAML" ,
      "url" :  "https://agent-connect.ai/agents/travel/weather/api_files/booking-interface.yaml" ,
      "description" :  "YAML file that provides OpenAPI for weather information subscription service."
    },
    {
      "@type" :  "ad:StructuredInterface" ,
      "protocol" :  "YAML" ,
      "url" :  "https://agent-connect.ai/agents/travel/weather/api_files/subscription-status-interface.yaml" ,
      "description" :  "YAML file that provides the OpenAPI for weather subscription status query service."
    },
    {
      "@type" :  "ad:NaturalLanguageInterface" ,
      "protocol" :  "YAML" ,
      "url" :  "https://agent-connect.ai/agents/travel/weather/api_files/nl-interface.yaml" ,
      "description" :  "Provides an interface for interacting with intelligent agents via natural language."
    }
  ],
"status_code" :  200 ,
"url" :  "https://agent-connect.ai/agents/travel/weather/ad.json"
}

In the above example, the agent description file uses the JSON-LD format. This format uses the common vocabulary provided by Schema.org, making the understanding of the data meaning more consistent and clear for different agents. In addition, WebAgent also supplements specific agent interaction specifications through custom extensions (such as ad fields).

By parsing these fields, AI basically understands how to use this WebAgent. ad:interfaces In the list, we see several OpenAPI YAML file links, one of which weather-info.yaml Provides the API definition of "weather query service".

After the AI ​​obtains this YAML file, it can know which HTTP interfaces the weather agent supports, such as GET /weather?city={city name} Such an endpoint, and the format of the returned weather information data. If AI wants to query the weather, it only needs to construct an HTTP request according to the OpenAPI specification.

Interaction example:  Assuming the description file tells us that the weather agent has an interface for obtaining weather, the AI ​​can perform the following steps:

1. Read the description file to determine whether DID authentication is required (if security If required) and find the interface definition that provides weather queries.
2. According to the interface definition, a specific request is formed. For example, "Query the current weather in Hangzhou" corresponds to a GET request:GET https://agent-weather.xyz/api/weather?city=Hangzhou(This URL is just an example, the actual endpoint needs to be determined according to the interface definition file).
3. Add the authentication information to the request header (see the next section for details), and then send the request to the WebAgent.
4. The WebAgent returns structured weather data (e.g., temperature, humidity, weather conditions, etc. in JSON format), which is directly parsed by the AI ​​and can be used for subsequent reasoning or presented to the user.

Through the description file, AI can gain a deeper understanding of the capabilities provided by WebAgent layer by layer: first discover which interfaces are available, then obtain API documents based on the interface description, and finally call specific APIs to complete the task. This design allows AI to autonomously crawl all machine-readable resources provided by WebAgent and complete complex tasks step by step without human intervention.

Authentication mechanism for accessing WebAgent

Considering that WebAgent is aimed at automated AI visitors, open interfaces may face problems of malicious abuse or excessive calls, such as the recent criticism of large model training companies for their crazy crawling of website data.

Therefore, WebAgent usually requires the caller to provide authentication to ensure that the request is from a trusted subject and to manage permissions or quotas. This is similar to the password, API Key or OAuth authorization required when humans access a website.

However, WebAgent adopts a solution that is more suitable for the intelligent agent ecosystem: an authentication mechanism based on  the W3C DID (Decentralized Identifier)  ​​standard. It allows intelligent agents to use their own DID as an identity to access WebAgent, without requiring each intelligent agent to register an additional account on the other party's system.

AgentNetworkProtocol (ANP) proposes a dedicated DID method—— did:wba (WebAgent DID). According to the ANP authentication specification (https://github.com/agent-network-protocol/AgentNetworkProtocol/blob/main/03-did%3Awba-method-design-specification.md) (DID: WBA method specification),did:wba It uses the existing Web infrastructure to implement decentralized identity authentication, which is designed for authentication between agents. In simple terms, each agent (or entity using WebAgent) can have a did:wba: Identifiers that begin with , for example:

did:wba:example.com:agent:weather123

This identifier is similar to an email address or user ID, and contains domain name information. In the WebAgent authentication system, it is a bit like "the other party only needs to know your DID to confirm your identity through the existing network infrastructure", without the need for each agent to register an additional account on the other party's system. This greatly simplifies cross-platform AI-Agent collaboration.

did:wba The identity authentication process can be summarized as follows:

• DID document publishing:  Each DID corresponds to a resolvable DID Document, which contains verification materials such as public keys.did:wba The method specifies that the DID document is hosted on a specific path of the corresponding domain name, which is convenient for retrieval via HTTP. For example, DID did:wba:agent-did.com:user:alice Its DID document (which will contain Alice's public key) can be parsed.
• Request with signature:  When an AI agent accesses a WebAgent, if the WebAgent requires DID authentication, it will ask the other party to provide a signature to prove the identity during the request process. Specifically, the requester uses its own private key to sign a challenge string or request content, and attaches the signature to the HTTP request header (such as Authorization)
• Verification process:  After receiving the request, the WebAgent server will extract the public key from the corresponding DID document based on the DID provided by the requester, verify whether the signature is valid, and thus confirm the identity of the requester. If the verification is successful, it is considered that the request comes from the entity represented by the DID, and then access or service is granted according to the policy. If the verification fails, a 401 unauthorized error is returned.

Since DID is decentralized, any two intelligent entities can trust each other without relying on a centralized user database . This is a bit like the email system - you don't need to register an account on the other party's email server to send an email to the other party, as long as the other party can find your domain name through DNS and trust the corresponding public key.did:wba The emergence of DID makes identity authentication in the WebAgent network flexible and platform-free. Compared with traditional OAuth or API Key solutions, DID is more suitable for large-scale, multi-agent intelligent ecosystems, because each Agent can have its own identity, authenticate in one place, and use it everywhere [Of course, in order to be compatible with existing systems, WebAgent may also support traditional API Key and other methods, but DID is obviously a future-oriented solution].

Anti-abuse and permission control:  By requiring the caller to provide a DID identity, WebAgent can:

• Assign call quotas and permission levels to each DID. For example, some sensitive operations can only be called by DIDs with a certain trust level.
• When problems arise, they can be traced back to specific DID identities, rather than being helpless in the face of anonymous crawler traffic.
• A reputation system is established between intelligent entities. DIDs with bad behavior can be blacklisted, while DIDs with good records may enjoy higher permissions.

It is worth noting that in order to lower the threshold for developers, the ANP community also provides a public DID for testing . This DID is did:wba:agent-did.com:test:public, the corresponding DID document and private key have been made public for testing and experiencing the WebAgent authentication process. Anyone can use this pair of public and private keys to try to call a WebAgent that requires authentication (only for testing, not for sensitive operations such as real transactions).

In actual development, we can quickly verify the WebAgent interface call process by introducing this set of public test DIDs. use_did_test_public The DID document for this test DID is provided in the directory (did.json) and the private key file (key-1_private.pem) and demonstrates how to use it. The code example in the next section will also show how to use the test identity to call the WebAgent interface.

Tools to access ANP WebAgent

We have developed a tool for interacting with ANP WebAgent, ANP Explorer, available at: https://service.agent-network-protocol.com/anp-explorer/.

It has two functions:

• It can be used as an AI personal assistant, and users can interact with the personal assistant through natural language. The personal assistant calls the WebAgent interface through the WebAgent description document and returns the results.
• Provides an exploration tool for agent description documents. You can enter the URL of the agent description document and then pull the document and other documents linked in the document.

This tool uses ANP to publicly test DID (did:wba:agent-did.com:test:public) Access WebAgent.

Code example: How to access the ANP protocol

After understanding the discovery, description, and authentication mechanisms of WebAgent, let's look at an actual code example to demonstrate how to use simple code to enable your AI Agent to access WebAgent. The code has been open sourced: anp-examples (https://github.com/agent-network-protocol/anp-examples). This code is the ANP Explorer code above.

The good news is that this process is very simple - as the community said: " Only a prompt word (Prompt) and an HTTP function " can communicate with any WebAgent.

Let’s look at the prompt words first. The code address is https://github.com/agent-network-protocol/anp-examples/blob/main/anp_examples/simple_example.py:

1. You will receive a starting URL ({{initial_url}}), which is a description file for the agent.
2. You need to understand the structure, functions and API usage of the agent description file.
3. You need to constantly discover and access new URLs and API endpoints like a web crawler.
4. You can use anp_tool to get the content of any URL.
5. The tool can handle responses in a variety of formats, including:
   - JSON format: will be parsed directly into a JSON object.
   - YAML format: will return text content and you need to analyze its structure.
   - Other text formats: The original text content will be returned.
6. Read each document to find task-related information or API endpoints.
7. You need to decide the crawling path by yourself and don’t wait for user instructions.
8. Note: You can crawl up to 10 URLs. You must end the crawl after exceeding this limit.
......

This description instructs the AI ​​model to act like a crawler, continuously calling the anp_tool tool to crawl the documents or interfaces of the intelligent agent, and repeating this process until it obtains the required information.

Next comes the HTTP functions section.anp_tool.py Defines an asynchronous method execute(...), whose role is to perform the actual HTTP request and handle the authentication logic.

For specific code, please refer to: https://github.com/agent-network-protocol/anp-examples/blob/main/anp_examples/anp_tool.py

It can be seen that with the help of ANPTool, the communication process between an AI Agent and a WebAgent is as simple as a normal HTTP call. For an intelligent agent based on a large model, it only needs to add such a tool with ANP function to its tool list and let the model know how to use it through prompts, and it can complete cross-agent operations autonomously.

For example, when a dialog agent with ANP capability receives a user's question "Book a hotel in Shanghai for me tomorrow", it can first use ANPTool to find the hotel booking agent -> obtain its description and interface -> complete the booking according to the process and return the result to the user. All this is due to the standardized, self-describing interface provided by WebAgent and the simple design of the ANP protocol.

Building a data network specifically for AI access

With the launch of the first WebAgent, we see a new form of the future Internet: a data service network that does not require an interface and is only open to AI . The construction of this network is based on the three key elements introduced in this article:

• Agent discovery : through standard discovery protocols and .well-known Entry, AI can automatically index intelligent services on the Internet without the need for manual directories.
• Agent description : Each WebAgent contains its own functional description and interface specifications, enabling AI to autonomously understand how to call services and form a highly self-describing network.
• Identity authentication : Through the decentralized identity (DID) mechanism, the security and credibility of network interactions are ensured, cross-platform permission management is supported, and the healthy operation of the network is guaranteed.

This “AI Data Network” has some notable properties:

• Designed for AI, UI-less : Network services no longer provide pages or graphical interfaces directly to humans, and all interactions occur between intelligent agents. This eliminates the need for traditional human-computer interfaces, reduces system complexity, and enables network services to focus entirely on the data itself, focusing on providing an efficient data exchange environment for AI.
• Semi-structured data : The data provided by WebAgent is mainly in structured format (such as JSON, YAML), which is convenient for AI agents to quickly parse and process; at the same time, natural language is partially used to carry personalized descriptive information to help the agent better understand the service content or express personalized characteristics.
• Self-description capability : WebAgent comes with a standardized description file that details the services it provides, interface definitions, and calling methods. When discovering services, the AI ​​agent can autonomously understand the specific functions and interaction methods of the services without human intervention or additional interface documentation.
• Safe and reliable : Considering that WebAgent is aimed at automated AI visitors, open interfaces may face the risk of malicious abuse or excessive calls (such as the recent widespread criticism of the crazy crawling of web page data by large model companies). Therefore, WebAgent has built-in authentication and permission management mechanisms based on decentralized identity (DID) to effectively avoid data abuse and ensure interface security. Through identity tracking and access authorization, a trustworthy intelligent network environment is established.

It is conceivable that in the near future, data and services from all walks of life may be open to AI in the form of WebAgents: weather, news, knowledge bases, trading platforms, merchants... AI will become the main "user", completing tasks that we can only complete through apps or web pages today directly through network interfaces. Human users will collaborate with AI and let AI call these WebAgents to complete complex goals.