With the rapid development of big model technology, the engineering of AI applications has put forward many requirements for the underlying infrastructure, especially in terms of security, efficiency, performance, etc. Among them, the AI ​​gateway is one of the most important AI infrastructure components.

The AI ​​Gateway is a deep evolution of the traditional API Gateway for large model scenarios. While ensuring the basic gateway capabilities, it has also made special enhancements to the characteristics of AI services:

• Adaptation to scenario characteristics : Specially optimizes transmission requirements such as long connections, high concurrency, and large bandwidth, and adapts to the high latency characteristics of large model services.
• Intelligent traffic management : supports multi-model dynamic routing, intelligent load balancing, API Key rotation scheduling, and semantic request caching.
• Security and compliance assurance : built-in content security filtering, token quota management, multi-level current limiting and circuit breaking, and other security capabilities.
• Fine-grained cost control : Provides operation and maintenance tools such as call audit analysis, grayscale traffic distribution, and automatic retry of failed requests.

By unifying the access layer protocol, the AI ​​Gateway helps developers achieve efficient integration and management of multi-source AI services, reducing access and operation and maintenance costs in complex scenarios. For a more comprehensive description of the core capabilities and usage scenarios of the AI ​​Gateway, please refer to the following two articles:

The core capabilities of the AI ​​Gateway are still in the early stages of definition, but they are inseparable from the core of rapid integration of AI Agent and LLM API.

Wangchen & Chengtan, WeChat Official Account: HigressAI 10 Basic Capabilities a Gateway Needs to Have

These eight usage scenarios are the most frequently used ones that we have summarized in the process of serving open source and commercial users. As the capabilities of the AI ​​gateway are expanded and enhanced, the usage scenarios are also gradually enriched.

Wangchen & Jiyuan, WeChat official account: Higress has sorted out 8 common application scenarios of AI gateways for you

The Higress open source site has added a sub-site specifically for AI scenarios on the original main site, and provides a Chinese version and an international version (Beta). The international version is used to serve global developers.

 In addition to providing common best practices (article format)/community/enterprise version/GitHub/ documentation and other functions, Higress.ai has specially designed a [scenario experience] for quickly experiencing the AI ​​gateway , and provides two ways of experience: open source experience and cloud experience. At the same time, we will launch the latest capabilities of the Higress  AI gateway on this site . For example, we are about to launch the AI ​​Guideline prompt function, and developers can quickly convert Nginx/Kong's Lua plug-in into a Higress Wasm plug-in based on AI programming tools such as Tongyi Lingma/Cursor.

After Higress.ai  goes online, you may be concerned about the following issues:

01

What is the difference between Higress.ai and Higress.cn?

Higress is a cloud-native API gateway. Its core is based on Istio and Envoy. It integrates traffic gateway, microservice gateway, security gateway and AI gateway into one. Wasm plug-ins can be written in Go/Rust/JS, etc. It provides dozens of ready-made general plug-ins and a ready-to-use console.

Higress.cn is  the main site of Higress. As the official technical portal and one-stop resource platform of Higress, it focuses on providing developers with core capability demonstrations, open source ecosystem support and best practices for enterprise users related to the Higress technology stack. 

Among them, AI gateway is a key component of modern AI infrastructure, and its technological evolution is deeply coupled with the ecological development of large language models. In the LLM technology stack, the continuous emergence of new technologies such as retrieval enhancement generation (RAG), agent, and MCP protocol has opened up multi-dimensional technological evolution directions for AI gateways in terms of protocol optimization, traffic management, model scheduling, and other dimensions.

In order to better show developers the richness of AI gateway content, Higress.ai came into being, aiming to provide an independent channel for experiencing AI gateway and display for typical AI application scenarios such as Agent development framework integration and LLM API governance. At the same time, Higress.ai will also showcase Higress's exploration of AI gateways, and work with AI developers to define the technical direction of the next generation of AI native gateways. In addition, Higress.ai will serve as our starting point for serving global AI developers.

It should be noted that the AI ​​gateway is not a new form independent of the API gateway. It is essentially also an API gateway. The difference is that it has been specially extended to meet the new requirements of AI scenarios. It is both a continuation of the API gateway and an evolution of the API gateway.

Of course not.

In the AI ​​era,  both Agents and large models have put forward more requirements on the access layer to avoid the "burden" of services. This has brought a historical development opportunity to AI gateways.

As early as last June when we released v1.4, we open-sourced many AI gateway capabilities. This was not a sudden idea after the accelerated development of large models during the Spring Festival. Further reading: Open-source AI gateway capabilities in June last year .

We believe that AI workloads and classic workloads will continue to merge to unleash the unlimited capabilities of AI and form unified management at the access layer. Therefore, Higress continues to focus on traffic gateways, microservice gateways, and security gateways to improve capabilities and experience.

At the traffic gateway level, Higress can be used as the Ingress entry gateway for the K8s cluster, and is compatible with a large number of K8s Nginx Ingress annotations, allowing for quick and smooth migration from K8s Nginx Ingress to Higress.

At the microservice gateway level, Higress can connect to various types of registry centers to discover service configuration routes, such as Nacos, ZooKeeper, Consul, Eureka, etc., and deeply integrates microservice technology stacks such as Dubbo, Nacos, and Sentinel. Compared with traditional Java-based microservice gateways, Higress based on the Envoy C++ gateway kernel can show better performance, significantly reduce resource utilization, and reduce costs.

At the security gateway level, Higress provides WAF capabilities and supports multiple authentication strategies, such as key-auth, hmac-auth, jwt-auth, basic-auth, OIDC, etc.

Higress's traffic gateway, microservice gateway, security gateway, and AI gateway all provide business-enhanced cloud services. The cloud service product on Alibaba Cloud is [API Gateway].

Higress.ai currently provides four experience scenarios: multi-model proxy, content security, semantic cache, and token current limiting . You can experience them with one click through the navigation bar at the top of the homepage.

Or you can click [Scene Experience] to enter the experience special page.

Higress.ai scene experience provides two ways:

• Cloud experience: Developers can log in to their Alibaba Cloud account and use the free quota to experience the AI ​​gateway function through the "Yunqi Lab" on the AIP gateway .
• Open source experience: Developers can deploy the AI ​​gateway with one click through the local Docker environment and experience the role of the AI ​​network in multiple scenarios.

Higress.ai's overall interface and function display are still being improved. Our next plan is:

• International version: All content of the site will be adapted for internationalization, and official overseas social media and technical community accounts will be enabled. Everyone is welcome to follow us.
• Online plugin editor: Developers can complete plugin development without configuring a local compilation environment. Based on AI programming capabilities, we will implement one-stop plugin generation services such as natural language plugin code generation, online compilation and debugging, and AI assistant problem solving in this editor to help you quickly produce plugins based on Higress.
• MCP Server: Higress.ai will be launched soon, converting the API capabilities of the backend service into MCP Server capabilities, thereby supporting a variety of MCP Client call scenarios and allowing API capabilities to be better used by AI tools. The official will support several mainstream applications, and developers can also freely access other applications.
• Best Practices Display: Covers  the entire life cycle of Agent development framework integration and LLM API governance, and outputs a series of best practices, including architecture design reference solutions, performance tuning suggestions, etc.
• More experience scenarios: Based on the 10+ plug-in functions in the open source version and the user scenarios in the commercial version, more experience scenarios will be launched; if you have any innovative practices on the open source version, please contact us for joint construction. (wx: zjjxg2018, note Higress)

Lend a hand

? Higress is an open-source high-performance gateway developed by Alibaba Cloud. It is used to deploy Web applications and large-scale model applications and provides commercial version services. Search for "API Gateway" on the Alibaba Cloud official website.

• Higress official website:
  https://higress.cn/
• API Gateway official website:
  https://www.aliyun.com/product/apigateway

• The latest progress in AI at home and abroad, not limited to technology, products and practices
• Higress's latest open source progress and customer practices
• Higress official documentation, tutorials, and the latest information on marketing activities

• WeChat: zjjxg2018
• Group: 107690002780

AI Gateway Agent LLMs Best Practices

The consumption of large model tokens may be a confusing account

After the big model made API popular, I share my practice of authentication and authorization for API security

Seamless switching of large models, QwQ-32B and DeepSeek-R1 are all required