EAS multi-node distributed inference enables you to deploy large MoE models such as DeepSeek 671B that exceed the capacity of a single device.
Prerequisites
Official SGLang and vLLM images from EAS or Model Gallery natively support distributed inference. To use a custom image, ensure it meets the networking requirements of your distributed inference framework and follows standard distributed processing patterns. For details, see How it works.
How it works
Key concepts and implementation principles for distributed inference:
Instance unit
Distributed inference introduces instance units (referred to as Unit in this topic). Instances within a Unit use high-performance network communication and parallelism strategies such as tensor parallelism (TP) and pipeline parallelism (PP) to process each request. Instances in a Unit are stateful. Units are fully symmetric and stateless relative to each other.
Instance ID
Each instance in a Unit receives a sequential instance ID through an environment variable. For the list of environment variables, see Appendix. Use these IDs to assign different tasks to instances.
Traffic handling
By default, each Unit handles traffic only through instance 0 (RANK_ID = 0). The system routes user traffic to instance 0 of each Unit using service discovery. Each Unit processes requests in a distributed manner and handles traffic independently.
Rolling updates
During a rolling update, a Unit is rebuilt as a whole. All instances in the new Unit start in parallel. After all new instances are ready, the system removes traffic from the old Unit and deletes its instances.
Lifecycle
Unit rebuild
When a Unit is recreated, all instances in the old Unit are deleted in parallel and all instances in the new Unit are created in parallel. No special handling occurs based on instance ID.
Instance recreation
By default, each instance in a Unit shares the lifecycle of instance 0. When instance 0 is recreated, all other instances in the Unit are also recreated. When a non-zero instance is recreated, other instances remain unaffected.
Distributed fault tolerance
-
Instance failure handling
-
When the system detects a failure in any instance of a distributed service, it automatically restarts all instances in the Unit.
-
This prevents cluster state inconsistency caused by single-point failures and ensures all instances start with a clean, consistent environment.
-
-
Coordinated recovery
-
After instances restart, the system waits for all instances in the Unit to reach a consistent ready state using a synchronization barrier. Business processes start only after all instances are ready.
-
This serves the following purposes:
-
Prevents NCCL communication group failures caused by inconsistent instance states.
-
Ensures strict startup synchronization across all nodes in a distributed inference task.
-
-
Distributed fault tolerance is disabled by default. To enable it, set the unit.guard parameter as shown in the following example:
{
"unit": {
"size": 2,
"guard": true
}
}Procedure
Deploy using Model Gallery
Model Gallery is recommended for multi-node distributed deployment.
-
Distributed inference is supported only when the inference engine is SGLang or vLLM.
-
Select deployment resources that match your chosen template.
On the model card, click Deploy. Then, click the edit icon next to the deployment template. In the dialog box that opens, select Distributed.
Deploy using a custom image
-
Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).
-
Create a service: On the Inference Service tab, click Deploy Service. Select Custom Model Deployment> Custom Deployment.
-
Update a service: On the Inference Service tab, find your target service in the service list. In the Actions column, click Update.
-
-
In the parameter configuration form, set the following parameters. For more details, see Custom Deployment.
-
In the Environment Information section, configure the runtime image and start command:
-
Image Configuration: In the Alibaba Cloud Image list, select a vllm or sglang image.
-
Command: After selecting an image, the system sets the start command automatically. Do not modify it.
-
-
In the Features area, turn on the Distributed Inference switch and configure the following key parameters:
Parameter
Description
Number of Machines for Single-Replica Deployment
Number of machines for a single model inference instance. The minimum value is 2.
RDMA Network
Enable RDMA to ensure high-speed network connections between machines.
NoteRDMA is available only for services deployed on Lingjun resources.
-
-
After configuring parameters, click Deploy or Update.
Appendix
When deploying a distributed inference service, configure networking for frameworks such as Torch Distributed or Ray. If you enable VPC or RDMA, each instance has multiple NICs. Specify which NIC to use for inter-instance communication.
-
RDMA enabled: Uses RDMA NIC (net0) by default.
-
RDMA disabled: Uses VPC NIC (eth1) configured by user.
Pass these settings as environment variables in your start command:
|
Environment Variable Name |
Description |
Example Value |
|
RANK_ID |
Instance ID, starting from 0 and incrementing by 1. |
0 |
|
COMM_IFNAME |
NIC for inter-instance communication:
|
net0 |
|
RANK_IP |
IP address for inter-instance communication, corresponding to the NIC specified by COMM_IFNAME. |
11.*.*.* |
|
MASTER_ADDRESS |
IP address of instance 0 (RANK_ID = 0), corresponding to the NIC specified by COMM_IFNAME for instance 0. |
11.*.*.* |