Huolala Dolphin Platform's Large Language Model Distributed Deployment Practice Based on LWS

Dolphin Platform currently uses K8S Deployment to deploy model services, where a Pod deploys a complete model instance and runs on a GPU physical machine. However, when deploying large language models, due to the limited arithmetic resources of physical nodes, a single physical machine cannot independently complete the deployment of large language models, and it is necessary to introduce a distributed framework for large language models, using strategies such as pipelined parallelism (PP) and tensor parallelism (TP), and utilizing multiple Pods to deploy large language models distributedly on multiple physical nodes. However, the Deployment-based model of hosting Pods suffers from the following problems in realizing this scheme:

1. In the large language model distributed deployment framework, different nodes have different startup commands, while the Deployment-hosted Pods have the same startup command, which cannot meet the requirements.

2. The large language model Distributed Deployment framework requires the master node IP to be fixed so that other nodes can connect and build node groups. However, Deployment deploys Pods with unfixed IP addresses, and the entire distributed framework fails to operate normally after the model service is restarted.

3. Pod groups composed of multiple Pods require overall scaling, rolling updates, and failure recovery capability, which Deployment does not support.

For the above problems, only the existing K8S Deployment capabilities can not meet the requirements of a large language model distributed deployment, so we must seek other solutions. The following will introduce how the Dolphin platform is based on K8S to achieve a large language model of multi-machine distributed deployment scheme and the principle behind it.

After research, we found that LeaderWorkerSet(LWS) can solve the AI/ML multi-machine distributed deployment problem. The Dolphin Platform realizes the K8S-based distributed deployment capability of the Dolphin Platform for large language models by introducing LWS with large language model distributed reasoning frameworks (e.g., Vllm, SGLang, LmDeploy, etc.) and deploying the large language models on multiple physical machines.

PodGroup as a whole unit: multiple Pods form a PodGroup, each Pod in the Group has a unique index from 0 to n-1 and has the same lifecycle, and all Pods have a globally unique network address identifier.

PodGroup supports multiple templates: A PodGroup is a group consisting of a single LeaderPod and multiple WorkerPods, allowing LeaderPods and WorkerPods to specify different templates (startup commands, resources, etc.).

PodGroup Expansion and Reduction: Supports the creation of multiple copies of the above Groups to achieve rapid expansion and reduction of capacity.

PodGroup Rolling Updates: PodGroups are upgraded as a whole unit (i.e., Pods within a Group are updated together).

PodGroup Failure Recovery: If a Pod in a Group fails, all Pods in the Group will be recreated.

In the following, we will briefly introduce the implementation principle of LWS features.

In a PodGroup, LeaderPod and WorkerPod are created by the corresponding LeaderStatefulSet and WorkerStatefulSet, respectively, and are associated with each other through labels, and the LeaderStatefulSet and WorkerStatefulSet can be respectively configured with different templates, such as startup commands, running mirrors, resource configuration, etc., thus successfully realizing the ability of LeaderPod and WorkerPod multi-models within a PodGroup.

In a large language model distributed deployment, LeaderPod must have a fixed network identity in order for WorkerPod to communicate. LWS uses a combination of StatefulSet and HeadlessService to realize the ability to fix the network identity of Pods within a Pod node group.

StatefulSet: It is a workload used by Kubernetes to manage stateful applications. It provides stateful applications with the ability to be deployed, scaled, and managed in an orderly manner. Unlike other controllers such as Deployment and DaemonSet, StatefulSet is more suited to stateful applications that require stable network identifiers and persistent storage, such as databases (e.g., MySQL, MongoDB, etc.), distributed caches (e.g., Redis clusters), and so on. StatefulSet generates stable Pod names for each Pod. For example, if you create a StatefulSet named haitun, it generates Pods named haitun-0, haitun-1. The names of these Pods remain the same throughout their lifecycle, and the names do not change even if the Pods are rescheduled or restarted.

The expansion and contraction mechanism of LWS is based on the cooperative operation of LeaderStatefulSet and WorkerStatefulSet. When a user modifies the spec.replicas field of the LWS, the LWS Controller senses the change and triggers the expansion and contraction process:

maxSurge: how many additional replicas can be deployed during the update, i.e., how many additional new Pods can be created.

maxUnavailable: indicates the number of unavailable replicas allowed during the update, i.e., the number of old Pods that can exit first during the update.

At present, Dolphin Platform has supported all the core capabilities of the AI development process, and the large language model application has also supported the training and deployment of large language model inference. And we will further improve the platform capabilities in the following aspects:

Improve the large language model application market: Unify the management of large language models on the Dolphin platform, and provide all-around support for rapid fine-tuning training, deployment, experience, and API access of large language models.

Distributed training of large language models: Introducing the industry's advanced large language model fine-tuning training framework, combining high-speed network transmission (RDMA) and high-performance file storage technology to improve the distributed fine-tuning training system for large language models.

Arithmetic Resource Utilization Improvement: In-depth optimization of arithmetic resource allocation and scheduling mechanism, focusing on efficient allocation and utilization of arithmetic resources for large language models, and improving overall arithmetic resource utilization efficiency.

Platform stability construction: build a multi-region and multi-cluster architecture to realize the multi-channel (multi-live) capability of algorithm services and improve the stability of the platform.