DocsICP FilingConsole
官方文档
首页

Deploy multi-node distributed inference services

更新时间：
复制 MD 格式
产品详情
我的收藏

Deploy Qwen3-32B across ACK nodes with LeaderWorkerSet (LWS), using vLLM or SGLang as the inference backend.

Background

Why multi-node inference

Large language models (LLMs) often exceed the memory of a single GPU. Multi-node inference splits the model across GPUs using one or more parallelization strategies:

Strategy How it works Best for
Data Parallelism (DP) Each GPU holds a full model copy and processes a different batch. Scaling throughput for smaller models
Tensor Parallelism (TP) Weight matrices are split across GPUs; each GPU computes on its slice. Large models that don't fit in a single GPU
Pipeline Parallelism (PP) Model layers run on different GPUs in a pipeline. Very deep models
Expert Parallelism (EP) For Mixture-of-Experts (MoE) models: experts are distributed across GPUs, and requests route to the relevant GPU. MoE architectures like Mixtral

This guide uses TP size 2, splitting the Qwen3-32B weights across two GPUs on two nodes.

How LWS maps to the parallelism configuration

LeaderWorkerSet organizes Pods into groups. In this setup:

  • The leader Pod runs the inference server as the Ray head node (vLLM) or primary server (SGLang).

  • The worker Pod holds the second GPU shard and communicates with the leader during inference.

The size: 2 in the LWS spec maps to the TP size: --tensor-parallel-size 2 (vLLM) or --tp 2 (SGLang). Each Pod runs on a separate GPU node.

image.png

Qwen3-32B

Qwen3-32B is a 32.8B-parameter dense model optimized for reasoning and conversational tasks:

  • Context window: 32K tokens (131K with YaRN)

  • Multilingual: 100+ languages

  • Capabilities: Reasoning, math, code generation, instruction following, multi-turn dialog, and agent tool use

See the blog, GitHub, and documentation.

vLLM

vLLM is a fast LLM inference library with PagedAttention-based KV cache management, continuous batching, speculative decoding, and CUDA/HIP graph acceleration. It supports TP, PP, DP, and EP parallelism on NVIDIA, AMD, and Intel GPUs, and exposes an OpenAI-compatible API. See vLLM GitHub.

SGLang

SGLang combines a high-performance inference backend with a flexible frontend for LLM and multimodal workloads. Backend features: RadixAttention (prefix cache), PagedAttention, continuous batching, speculative decoding, PD separation, and multi-LoRA batching. Supports TP, PP, DP, and EP parallelism, with quantization formats including FP8, INT4, AWQ, and GPTQ. See SGLang GitHub.

Prerequisites

Before you begin:

  • An ACK managed cluster (Kubernetes 1.28+) with two or more GPU-accelerated nodes, each with more than 32 GB of GPU memory.

    Recommended instance type: ecs.gn8is.4xlarge. See GPU-accelerated compute-optimized instance family gn8is.

  • LeaderWorkerSet (LWS) v0.6.0+ installed in your cluster. To install via the ACK console:

    1. Log on to the ACK console.

    2. In the left navigation pane, click Clusters, then click your cluster name.

    3. In the left navigation pane, choose Applications > Helm. On the Helm page, click Deploy.

    4. In Basic Information, enter the Application Name (lws) and Namespace (lws-system). In Chart, find lws and click Next.

    5. In Parameters, select the latest Chart Version and click OK.

    image

Step 1: Prepare the Qwen3-32B model files

Download the model

Download Qwen3-32B from ModelScope using Git LFS.

If git-lfs is not installed, run yum install git-lfs or apt-get install git-lfs . See Installing Git Large File Storage.
git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/Qwen/Qwen3-32B.git
cd Qwen3-32B/
git lfs pull

Upload the model to OSS

Log on to the OSS console and note your bucket name. If you don't have one, create a bucket first. Upload the model files:

See Install ossutil.
ossutil mkdir oss://<your-bucket-name>/Qwen3-32B
ossutil cp -r ./Qwen3-32B oss://<your-bucket-name>/Qwen3-32B

Create a PV and PVC for the model

Create a persistent volume (PV) and persistent volume claim (PVC) named llm-model to make the model accessible to cluster Pods. See Create a PV and a PVC.

Option 1: ACK console

  1. Create a PV. In the ACK console, go to your cluster and choose Volumes > Persistent Volumes. Click Create and configure the following:

    Parameter Value
    PV Type OSS
    Volume Name llm-model
    Access Certificate Your AccessKey ID and AccessKey secret
    Bucket ID The OSS bucket you created
    OSS Path /Qwen3-32B

  2. Create a PVC. Go to Volumes > Persistent Volume Claims and click Create. Configure the following:

    Parameter Value
    PVC Type OSS
    Name llm-model
    Allocation Mode Existing Volumes
    Existing Volumes Click Select PV and select the PV you created

Option 2: kubectl

Create a file named llm-model.yaml with the following content:

apiVersion: v1
kind: Secret
metadata:
  name: oss-secret
stringData:
  akId: <your-oss-ak>      # The AccessKey ID used to access the OSS bucket.
  akSecret: <your-oss-sk>  # The AccessKey secret used to access the OSS bucket.
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: llm-model
  labels:
    alicloud-pvname: llm-model
spec:
  capacity:
    storage: 30Gi
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ossplugin.csi.alibabacloud.com
    volumeHandle: llm-model
    nodePublishSecretRef:
      name: oss-secret
      namespace: default
    volumeAttributes:
      bucket: <your-bucket-name>       # The bucket name.
      url: <your-bucket-endpoint>      # The endpoint, such as oss-cn-hangzhou-internal.aliyuncs.com.
      otherOpts: "-o umask=022 -o max_stat_cache_size=0 -o allow_other"
      path: <your-model-path>          # In this example, the path is /Qwen3-32B/.
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: llm-model
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 30Gi
  selector:
    matchLabels:
      alicloud-pvname: llm-model

Apply the manifest:

kubectl create -f llm-model.yaml

Step 2: Deploy the distributed inference service

Both backends use an LWS workload with size: 2 (one leader + one worker Pod, each on a separate GPU node) and a TP size of 2. The leader runs the inference server; the worker holds the second model shard.

Deploy with vLLM

  1. Create a file named vllm_multi.yaml:

    Expand to view the YAML template

    apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: vllm-multi-nodes
spec:
  replicas: 1
  leaderWorkerTemplate:
    size: 2
    restartPolicy: RecreateGroupOnPodRestart
    leaderTemplate:
      metadata:
        labels:
          role: leader
          # for prometheus to scrape
          alibabacloud.com/inference-workload: vllm-multi-nodes
          alibabacloud.com/inference_backend: vllm
      spec:
        volumes:
          - name: model
            persistentVolumeClaim:
              claimName: llm-model
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 15Gi
        containers:
          - name: vllm-leader
            image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:v0.10.0
            command:
              - sh
              - -c
              - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE); python3 -m vllm.entrypoints.openai.api_server --port 8000 --model /models/Qwen3-32B --tensor-parallel-size 2"
            resources:
              limits:
                nvidia.com/gpu: "1"
                memory: "24Gi"
                cpu: "8"
              requests:
                nvidia.com/gpu: "1"
                memory: "24Gi"
                cpu: "8"
            ports:
              - containerPort: 8000
                name: http
            readinessProbe:
              initialDelaySeconds: 30
              periodSeconds: 10
              tcpSocket:
                port: 8000
            volumeMounts:
              - mountPath: /models/Qwen3-32B
                name: model
              - mountPath: /dev/shm
                name: dshm
    workerTemplate:
      spec:
        volumes:
          - name: model
            persistentVolumeClaim:
              claimName: llm-model
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 15Gi
        containers:
          - name: vllm-worker
            image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:v0.10.0
            command:
              - sh
              - -c
              - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"
            resources:
              limits:
                nvidia.com/gpu: "1"
                memory: "24Gi"
                cpu: "8"
              requests:
                nvidia.com/gpu: "1"
                memory: "24Gi"
                cpu: "8"
            ports:
              - containerPort: 8000
            volumeMounts:
              - mountPath: /models/Qwen3-32B
                name: model
              - mountPath: /dev/shm
                name: dshm
---
apiVersion: v1
kind: Service
metadata:
  name: multi-nodes-service
spec:
  type: ClusterIP
  ports:
  - port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    alibabacloud.com/inference-workload: vllm-multi-nodes
    role: leader

  2. Deploy the service:

    kubectl create -f vllm_multi.yaml

Deploy with SGLang

  1. Create a file named sglang_multi.yaml:

    Expand to view the YAML template

    apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: sglang-multi-nodes
spec:
  replicas: 1
  leaderWorkerTemplate:
    size: 2
    restartPolicy: RecreateGroupOnPodRestart
    leaderTemplate:
      metadata:
        labels:
          role: leader
          # for prometheus to scrape
          alibabacloud.com/inference-workload: sglang-multi-nodes
          alibabacloud.com/inference_backend: sglang
      spec:
        containers:
          - name: sglang-leader
            image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
            command:
              - sh
              - -c
              - "python3 -m sglang.launch_server --model-path /models/Qwen3-32B --tp 2 --dist-init-addr $(LWS_LEADER_ADDRESS):20000 \
              --nnodes $(LWS_GROUP_SIZE) --node-rank $(LWS_WORKER_INDEX) --trust-remote-code --host 0.0.0.0 --port 8000 --enable-metrics"
            resources:
              limits:
                nvidia.com/gpu: "1"
                memory: "24Gi"
                cpu: "8"
              requests:
                nvidia.com/gpu: "1"
                memory: "24Gi"
                cpu: "8"
            ports:
              - containerPort: 8000
                name: http
            readinessProbe:
              tcpSocket:
                port: 8000
              initialDelaySeconds: 30
              periodSeconds: 10
            volumeMounts:
              - mountPath: /dev/shm
                name: dshm
              - mountPath: /models/Qwen3-32B
                name: model
        volumes:
            - name: model
              persistentVolumeClaim:
                claimName: llm-model
            - name: dshm
              emptyDir:
                medium: Memory
                sizeLimit: 15Gi
    workerTemplate:
      spec:
        containers:
          - name: sglang-worker
            image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
            command:
              - sh
              - -c
              - "python3 -m sglang.launch_server --model-path /models/Qwen3-32B --tp 2 --dist-init-addr $(LWS_LEADER_ADDRESS):20000 \
              --nnodes $(LWS_GROUP_SIZE) --node-rank $(LWS_WORKER_INDEX) --trust-remote-code"
            resources:
              limits:
                nvidia.com/gpu: "1"
                memory: "24Gi"
                cpu: "8"
              requests:
                nvidia.com/gpu: "1"
                memory: "24Gi"
                cpu: "8"
            volumeMounts:
              - mountPath: /dev/shm
                name: dshm
              - mountPath: /models/Qwen3-32B
                name: model
        volumes:
            - name: model
              persistentVolumeClaim:
                claimName: llm-model
            - name: dshm
              emptyDir:
                medium: Memory
                sizeLimit: 15Gi
---
apiVersion: v1
kind: Service
metadata:
  name: multi-nodes-service
spec:
  selector:
    alibabacloud.com/inference-workload: sglang-multi-nodes
    role: leader
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000

  2. Deploy the service:

    kubectl create -f sglang_multi.yaml

Step 3: Verify the inference service

Test with a sample request

Important

kubectl port-forward is for development and debugging only — it lacks the reliability, security, and scalability for production. Use Ingress for production access.

  1. Forward port 8000 to your local machine:

    kubectl port-forward svc/multi-nodes-service 8000:8000

    Expected output:

    Forwarding from 127.0.0.1:8000 -> 8000
Forwarding from [::1]:8000 -> 8000

  2. Send a test inference request:

    curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "/models/Qwen3-32B", "messages": [{"role": "user", "content": "Test it"}], "max_tokens": 30, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

    Expected response:

    {"id":"chatcmpl-ee6b347a8bd049f9a502669db0817938","object":"chat.completion","created":1753685847,"model":"/models/Qwen3-32B","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"<think>\nOkay, the user sent \"Test it\". I need to confirm their request first. They might be testing my functionality or want to see my reaction.","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":10,"total_tokens":40,"completion_tokens":30,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}

    This confirms the distributed inference service is running.

Next steps

  • Configure an Ingress for production access.

  • Enable autoscaling for the LWS workload to handle variable traffic.

  • Monitor GPU utilization and throughput with the Prometheus labels (alibabacloud.com/inference_backend) on the leader Pod.

Previous：Deploy standalone LLM inference servicesNext：Deploy an SGLang inference service with Prefill/Decode separation