In today's era of booming artificial intelligence, large language models ( LLMs ) have sprung up like mushrooms after a rain. They have shown their prowess in various fields of natural language processing, from text generation to intelligent question and answer, from machine translation to sentiment analysis. However, the numerous models also bring new challenges. The model API formats of different manufacturers vary greatly. When developers integrate multiple models, they need to spend a lot of time and energy to learn and adapt different interfaces, which undoubtedly increases the complexity and cost of development. The open source project litellm came into being. It aims to build a unified bridge for developers to cross the gap between different large language model APIs , making model calls easy and efficient.

1. Project Overview

litellm is an open source LLM calling framework carefully built by the BerriAI team. Its core mission is to solve the problem of inconsistent formats faced by calling model APIs of different manufacturers in the development of large language model applications . Through innovative design, litellm uses a unified OpenAI format to call more than 100 different large language models, including models from many well-known manufacturers such as Bedrock , Azure , OpenAI , Cohere , Anthropic , Ollama , Sagemaker , HuggingFace , Replicate , etc. This unified calling method greatly simplifies the development process. Developers no longer need to write adaptation code for each model separately , which greatly improves development efficiency.

2. Core Functions

1. Unified input and output formats

Litellm uses OpenAI 's API format as the standard to uniformly encapsulate all supported large language models. When calling different models, developers only need to construct requests in the OpenAI format without having to worry about the unique input requirements of each model. When generating text, whether calling OpenAI 's GPT-3.5-Turbo or Anthropic 's Claude , the request format remains consistent. In terms of responses, Litellm also ensures that the text responses of all models are presented in a unified, easily accessible format.

(II) Automatic request conversion

When developers send a request based on the unified OpenAI format, litellm will automatically convert it into the specific format required by each vendor's model API behind the scenes. This process is completely transparent to developers, and they do not need to understand the details of each model API . You can easily adapt to multiple models with one set of code.

3. Failover and load balancing between multiple models

In actual applications, models may fail, undergo maintenance, or respond slowly due to high concurrency. Litellm provides powerful failover and load balancing capabilities.

• Failover : Developers can set up multiple models as backup options. When an error occurs in the primary model, Litellm will automatically try to use the backup model to handle requests, ensuring service continuity.

  
  

• Load balancing : Litellm supports advanced routing strategies, such as the " least-busy " strategy. This strategy tracks the number of requests currently processed by each model in real time. When a new request arrives, it is assigned to the model that currently processes the least requests, thereby achieving a balanced distribution of load and improving the performance and response speed of the overall system.

  
  

4. Expenditure tracking and budget control

Using multiple large language models is often accompanied by cost issues, especially for enterprise users, cost control is crucial. Litellm provides comprehensive cost tracking and budget control functions.

• Cost tracking : Through the built-in tracking mechanism, litellm can record the cost of each model call in detail, including the number of tokens used, unit price and other information, so that users can clearly understand the cost of each operation.

  
  

• Budget control : Users can set budget limits for each project, API key, or model. When usage approaches or exceeds the budget, Litellm can issue an alert or automatically stop related requests to help users effectively manage costs and avoid unnecessary expenses.

  
  

5. Log Monitoring and Observability

Litellm supports integration with mainstream log monitoring and observability platforms, such as Lunary , Langfuse , Helicone , etc. Through simple configuration, Litellm can record detailed information of model calls , including request parameters, response results, call time, response time, error information, etc. These log data are very helpful for developers to troubleshoot, optimize performance, and understand system operation status.

3. Advantages and characteristics

1. Development efficiency has been greatly improved

One set of code can call all supported models, and developers do not need to write different interface codes for different models. When developing an intelligent customer service system, you may need to write adaptation codes for different models separately. Now with litellm , you only need to write the code once in a unified format, and you can easily switch to different models, which greatly shortens the development cycle. In addition, exception handling also adopts the OpenAI format, making the error handling logic more consistent and concise, further improving development efficiency.

2. More convenient operation and maintenance management

• Automatic retry and failover : Reduces the risk of service interruption due to model failure and ensures high availability of the system.

  
  

• Complete monitoring and logging : enables operation and maintenance personnel to understand the system operation status in real time and quickly locate and solve problems.

  
  

• Fine-grained cost control : enables enterprises to better manage resources and budgets and optimize cost structures. Through litellm proxy services, enterprises can also uniformly manage access to multiple models, set access control and rate limits, and ensure the security and stability of the system.

  
  

3. Rich and flexible functions

• Streaming output support : litellm supports streaming output, which is very useful in some scenarios with high real-time requirements, such as chatbots. Users can gradually receive and display content while the model generates text, improving the interactive experience.

  
  

• Custom callback : Provides custom callback function. Developers can insert custom code logic at different stages of model calling according to their own needs to achieve more flexible business needs.

  
  

• Observability platform support : Litellm also supports mainstream observability platforms, which facilitates integration with the company's existing monitoring and management systems and enhances the manageability and scalability of the system.

  
  

IV. Application Scenarios

1. Application development field

In the development of various applications, whether it is a web application, mobile application or desktop application, it may be necessary to integrate a large language model to implement intelligent functions. During the application development process, if you need to test the impact of different models on application performance and functions, you can use litellm to quickly switch models without modifying a large amount of code, greatly improving development and testing efficiency.

2. Enterprise service scenarios

For enterprises, litellm has a wide range of application value:

• Customer service : Enterprises can use litellm to integrate multiple models to build an intelligent customer service system, and distribute customer consultation requests to different models for processing through load balancing, thereby improving response speed and service quality.

  
  

• Data analysis and processing : Enterprises can use litellm to call different models to analyze large amounts of text data, such as sentiment analysis, topic classification, etc., and reasonably control costs through cost tracking and budget control functions.

  
  

• Digital transformation : During the digital transformation process, enterprises may need to combine different business processes with large language models. Litellm can serve as a unified model calling interface to simplify system architecture and improve development and deployment efficiency.

3. Scientific research and exploration scenarios

When conducting research related to natural language processing, researchers often need to compare and test multiple large language models to find the model or combination that best suits a specific research task. Litellm provides convenience for researchers, who can quickly call different models for experiments through a unified interface, reducing the operational complexity caused by different model interfaces. When studying new text generation algorithms, researchers can use Litellm to easily switch between different basic models for experiments, observe the impact of different models on the algorithm effect, accelerate the research process, and improve research efficiency.

5. Quick Use

1. Installation

The installation of litellm is very simple and can be completed using the pip package manager. Enter the following command in the command line:

pip install litellm

During the installation process, pip will automatically download and install litellm and its dependent libraries. After the installation is complete, you can import and use litellm in your Python project .

(II) Basic call examples

The following is a simple code example showing how to use litellm to call the model. Suppose we want to use the GPT - 3.5 - Turbo model to generate a piece of text:

from litellm import completionimport os # Set the API key. Here we take OpenAI's API key as an example os.environ["openai_api_key"] = "your_openai_api_key" # Call the model response = completion( model="gpt-3.5-turbo", messages=[{"content": "Please introduce the application of artificial intelligence in the medical field", "role": "user"}]) print(response.choices[0].message.content)

In the above code, we first import the completion function of litellm , and then set the OpenAI API key (replace it with the real key when actually using it). Next, we call the GPT - 3.5 - Turbo model through the completion function , passing in a list of messages containing user questions. In the response returned by the model, we extract and print the generated text content.

3. Using a proxy server

If you use it as a team and want to build a centralized LLM gateway to uniformly manage access to multiple models, you can use the proxy server function of litellm .

1.  Start the proxy server : First, start the proxy server. For example, to use the Hugging Face model:

litellm --model huggingface/bigcode/starcoder

After successful startup, the proxy server will listen to the specified port ( 4000 by default ).

2.  Code call settings : Then, when calling the OpenAI client in the code , set the api_key (you can set it to any value here, because the proxy server does not rely on this key verification) and base_url to the address of the proxy server:

import openaiclient = openai.OpenAI( api_key="anything", base_url="http://0.0.0.0:4000")response = client.chat.completions.create( model="huggingface/bigcode/starcoder", messages=[{"content": "Write a simple Python function", "role": "user"}])print(response.choices[0].message.content)

This way, teams can easily and centrally manage model access, track usage and costs, and set access controls, rate limits, and more.

VI. Conclusion

As an innovative large language model calling framework, litellm has brought many conveniences to developers, enterprises, and scientific researchers. It effectively solves the complexity of multi-model calling, significantly improves development efficiency, optimizes operation and maintenance management, and expands the application of large language models in different scenarios. Whether it is individual developers quickly building AI applications or enterprises integrating and managing multiple models on a large scale, litellm can be a powerful assistant.