This article will take you deep into Transformers, vLLM, Llama.cpp, SGLang, MLX and Ollama engines to help you find the most suitable tools and unleash the full potential of large language models!

1. Transformers Engine: The Almighty King in the NLP Field

Developer : Hugging Face

Core features : As the most popular open source NLP library, Transformers is known as the "Swiss Army Knife" in the field of NLP. It supports hundreds of pre-trained models, including well-known models such as GPT, BERT, and T5, and provides a one-stop solution from model loading, fine-tuning, to reasoning.

Significant advantages :

• Strong compatibility : Perfectly compatible with PyTorch and TensorFlow, providing developers with more choices.

• Ecosystem prosperity : With an active community, rich model library and complete documentation, both beginners and experts can benefit from it.

• Widely used : Suitable for various NLP tasks from academic research to industrial production.
  

• Applicable scenarios : When you need to quickly implement tasks such as text classification, generation, and translation, Transformers is the best choice, which can help you easily implement NLP applications.

  
  

2. vLLM Engine: Peak Performance of GPU Inference

Developer : UC Berkeley research team

Core features : vLLM focuses on large language model reasoning. With innovative memory management technologies (such as PagedAttention), it greatly improves GPU utilization and reasoning speed, making it a "performance monster" for GPU reasoning.

Significant advantages :

• Excellent performance : Extreme inference speed, able to meet large-scale deployment requirements.

• Memory Efficient : Efficient memory management, supporting larger model batch processing.

• Scene adaptation : Optimized for GPU, with excellent performance in high-concurrency scenes.
  

  Applicable scenarios : If you need to deploy a large language model in a production environment and pursue extreme performance, vLLM is undoubtedly the best choice. It can improve the model inference speed and reduce hardware costs.

  
  

3. Llama.cpp engine: lightweight pioneer on CPU

Developer : Community Project

Core features : Llama.cpp is implemented based on C++ and is specially designed for running Meta's LLaMA model. By optimizing calculation and memory management, it makes it possible to run large models on the CPU. It is the "lightweight king" on CPU devices.

Significant advantages :

• Lightweight : No GPU is required and it can run on ordinary CPU devices.

• Flexible deployment : suitable for resource-constrained environments, such as embedded devices and low-profile servers.

• Open source expansion : The open source feature makes it easy to expand and customize.
  

• Applicable scenarios : When the device has no GPU resources but needs to run a large language model, Llama.cpp is an ideal choice, allowing ordinary devices to experience the power of large language models.

  
  

4. SGLang Engine: A Potential Rising Star for Efficient Reasoning

Developer : Unknown

Core features : SGLang focuses on efficient reasoning and may use technologies such as sparse computing and distributed optimization to improve performance. Although it is full of mystery, it has unlimited potential.

Significant advantages :

• Scenario optimization : Deeply optimize for specific scenarios to significantly improve reasoning efficiency.

• Enterprise adaptation : Suitable for enterprise-level applications that require high-performance reasoning.
  

• Applicable scenarios : SGLang is worth a try when running large language models in a large-scale distributed environment. It is an important window for exploring future reasoning technologies.

  
  

MLX Engine: The Future of Efficient Computing

Developer : Unknown

Core features : MLX may be a machine learning framework optimized for large language models, focusing on efficient computing and reasoning, and is a "future star" in the field of efficient computing.

Significant advantages :

• Hardware adaptation : It may be optimized for specific hardware such as TPU or custom chips.

• Efficiency first : Applicable to scenarios that require extreme computing efficiency.
  

• Applicable scenarios : If you need to run a large language model on specific hardware, MLX is worth paying attention to. Its potential hardware optimization capabilities are expected to lead the future of efficient computing.

  
  

6. Ollama: A convenient choice for running large local models

Developer : Community Project

Core features : Ollama is a powerful tool for running large language models locally. It supports multiple models such as LLaMA and GPT, and simplifies the model deployment and operation process.

Significant advantages :

• Easy to use : easy to operate, suitable for individual users and developers.

• Local operation : No cloud resources are required, and the model can be run completely on the local device.

• Rich models : supports multiple models and is flexible to use.
  

• Applicable scenarios : If you want to test or run a large language model on your personal device, Ollama is an excellent choice, helping you get rid of cloud dependence and experience the charm of large models at any time 

7. Index comparison

1. Performance comparison

Summarize :

• vLLM  has the best inference performance on GPU and is suitable for large-scale models.

• Llama.cpp  and  Ollama  are suitable for running small and medium-sized models on CPU or low-end devices.

• SGLang  and  MLX  have great performance potential, but more practical verification is needed.

2.  Concurrency Comparison

Summarize :

• vLLM  performs best in high-concurrency scenarios and is suitable for production environments.

• Transformers  and  SGLang  are suitable for moderately concurrent tasks.

• Llama.cpp  and  Ollama  are more suitable for single-task or low-concurrency scenarios.

3.  Comparison of applicable scenarios

Summarize :

• Transformers  are the most versatile tool and suitable for most NLP tasks.

• vLLM  is the first choice for enterprise-level high-concurrency scenarios.

• Llama.cpp  and  Ollama  are suitable for individual developers or resource-constrained environments.

• SGLang  and  MLX  are suitable for scenarios that require high performance or specific hardware support.

4.  Hardware compatibility comparison

Summarize :

• Transformers  and  Ollama  are the most compatible and support multiple devices.

• vLLM  and  SGLang  require high-performance GPU or server.

• Llama.cpp  is suitable for low-end devices, while  MLX  requires specific hardware support.

8. Comparison of token output per second

1. Factors affecting performance

Before comparing TPS, it is necessary to clarify the key factors that affect performance:

• Hardware performance: GPU computing power, video memory bandwidth, video memory capacity, etc.

• Model size: The larger the number of parameters, the slower the inference speed.

• Batch Size: Larger batches can increase throughput but increase video memory usage.

• Engine optimization: Different engines have significant differences in performance in terms of memory management, computational optimization, etc.

  
  

2.  GPU Performance Comparison

The following is a comparison of the main parameters of A800, A100 and H100:

• H100  is the most powerful GPU currently available, suitable for high-throughput and high-concurrency scenarios.

• The performance of A100  and  A800  is similar, but A800 is mainly aimed at the Chinese market and complies with export control requirements.

3.  Engine TPS comparison

The following are the estimated TPS of each engine on different GPUs  (taking the LLaMA-13B model as an example):

illustrate:

• vLLM performs best on high-performance GPUs (such as H100), with TPS reaching 500-1000, far exceeding other engines.

• Transformers have medium performance and are suitable for general scenarios.

• Llama.cpp and Ollama have lower performance and are suitable for resource-constrained environments.

• The performance data of SGLang and MLX are less and need further testing.

9. Here is a brief introduction to Xinference installation

Xinference can be used on Linux, Windows, and MacOS via pip To install it. If you need to use Xinference for model reasoning, you can specify different engines according to different models.

If you want to be able to infer all supported models, you can install all required dependencies with the following command:

pipinstall "xinference[all]"

Remark

If you want to use GGML models, it is recommended to manually install the required dependencies according to the hardware you are using to take full advantage of the hardware acceleration capabilities. For more details, please refer to  the Llama.cpp engine  section.

If you just want to install the necessary dependencies, here are the detailed steps on how to do it.

pipinstall"xinference[transformers]"

pipinstall "xinference[vllm]"

pipinstallxinference

pipinstall'xinference[sglang]'

10. Xinference  environment variables