Load-Aware Routing Practice

1. (Optional) Deploy a sample inference service

This step provides a sample vLLM inference service that exposes Prometheus metrics (using DeepSeek-R1-Distill-Qwen-1.5B as an example, with the ECS instance type ecs.gn7i-c16g1.4xlarge). This service is used for load-aware routing metric collection and scheduling in subsequent steps. The verification test section also uses this sample model for benchmarking. If you have already deployed an inference service as described in the Prerequisites section, you can skip this step. You can also refer to Deploy a Qwen large model inference service.

1. Prepare the model and configure storage volumes. After downloading the model and uploading it to OSS, refer to Configure OSS storage volumes to configure PV and PVC (such as example-oss-swap) for the cluster, which the inference service mounts to access the model. Storing the model in OSS and pulling it via internal network avoids the long time required for public network downloads.

   # 1. Download the model
   GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B.git
   cd DeepSeek-R1-Distill-Qwen-1.5B
   git lfs pull
   
   # 2. Upload the model to OSS
   ossutil mkdir oss://<Your-Bucket-Name>/DeepSeek-R1-Distill-Qwen-1.5B
   ossutil cp -r ./DeepSeek-R1-Distill-Qwen-1.5B oss://<Your-Bucket-Name>/DeepSeek-R1-Distill-Qwen-1.5B

2. Create a Deployment and Service for the inference service in the ACK cluster. The following configuration starts an inference service using vLLM and declares Prometheus collection annotations on the Pod, exposing the 8000 port's /metrics as the metrics endpoint.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: deepseek-r1-distill-qwen-1.5b
     name: deepseek-r1-distill-qwen-1.5b
     namespace: default
   spec:
     replicas: 4
     selector:
       matchLabels:
         app: deepseek-r1-distill-qwen-1.5b
     template:
       metadata:
         labels:
           app: deepseek-r1-distill-qwen-1.5b
         annotations:
           prometheus.io/path: /metrics
           prometheus.io/port: "8000"
           prometheus.io/scrape: "true"
       spec:
         volumes:
           - name: model
             persistentVolumeClaim:
               claimName: example-oss-swap
           - name: dshm
             emptyDir:
               medium: Memory
               sizeLimit: 30Gi
         containers:
         - command:
           - sh
           - -c
           - vllm serve /models/DeepSeek-R1-Distill-Qwen-1.5B --port 8000 --trust-remote-code --served-model-name deepseek-r1-distill-qwen-1.5b --gpu-memory-utilization 0.95 --enforce-eager
           image: registry-cn-hangzhou.ack.aliyuncs.com/dev/vllm:0.10.0
           env:
           - name: SAFETENSORS_FAST_GPU_TRANSFER
             value: "0"
           - name: SAFETENSORS_MAX_HEADER_LENGTH
             value: "10000000"
           name: vllm
           ports:
           - containerPort: 8000
           readinessProbe:
             tcpSocket:
               port: 8000
             initialDelaySeconds: 30
             periodSeconds: 30
           resources:
             limits:
               nvidia.com/gpu: "1"
           volumeMounts:
             - mountPath: /models/
               name: model
             - mountPath: /dev/shm
               name: dshm
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: deepseek-r1-distill-qwen-1-5b-v1
   spec:
     type: ClusterIP
     ports:
     - port: 8000
       protocol: TCP
       targetPort: 8000
     selector:
       app: deepseek-r1-distill-qwen-1.5b

3. After the Pods are ready, exec into any Pod to verify that the metrics endpoint is properly exposed. The response should contain metrics such as vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, and vllm:num_requests_running.

   curl http://<Pod-IP>:8000/metrics

2. Create an ALB Extensible Edition instance

1. Log on to the ALB console. Select the China (Shanghai) region and click Create ALB.

2. On the buy page, complete the following configuration and click Create Now.

   • Region: Select China (Shanghai).

   • Instance Network Type: Select Internet.

   • VPC and Zone: Select VPC1, check Shanghai Zone B and Shanghai Zone F, then select VSW1 and VSW2.

   • IP Version: Select IPv4.

   • Edition (Instance Fee): Select Extensible Edition.

3. On the Confirm Order page, confirm the instance configuration details and click Activate Now.

3. Create a Service Extension and add the load-aware routing component

Metric collection requests are sent over HTTP 1.1 using IPv4 addresses only. IPv6 backends are not supported. The metric collection response body for a single backend is limited to 10 KB by default. Place the required metrics (vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, vllm:num_requests_running) within the first 10 KB of the metrics output. For additional details, see Limits.

1. In the Service Extension console, click Create Service Extension. In the Service Extension Configuration section, enter a Extension name such as ext-load-aware-routing.

2. Extension Type is set to Plug-in by default. From the Component name dropdown, select Load-Aware Routing. Complete the following configuration and click Create.

   • Collection Configuration:

     • Host: The Host request header value for metric collection requests. Leave this field empty in this example, which means the system uses the IP:Port of each backend in the server group for collection.

     • Path: The request path for metric collection. This example uses /metrics.

     • Response Timeout: Default is 5 seconds.

     • Interval: Set to 1 second.

   • Metric Configuration: Select all three: Total Queued Requests, GPU Cache Utilization, and Running Requests. Set the weights to 100, 90, and 80 respectively.

The load-aware routing component must be used with a Server or IP Address server group, and cannot be added to the same Service Extension as other components.

4. Create a server group and associate the Service Extension

Create a server group to host the backend inference replicas and associate it with the load-aware routing Service Extension created in Step 3.

1. In the Server Group console, select the China (Shanghai) region. Click Create Server Group, complete the following configuration, and click Create.

   • Server Group Type: Select IP Address.

   • Server Group Name: Enter a custom name such as sgp-load-aware.

   • VPC: Select VPC1.

   • Scheduling Algorithm: Use the default Weighted Round-robin. Load-aware routing only supports combination with weighted round-robin.

   • Disable Health Check.

     In this example, the backend Pods do not provide the interface required for ALB health checks. If health checks are enabled, backends would be determined unhealthy, causing load-aware routing to be bypassed. If your backends provide a health check interface, keep health checks enabled. Disabling is not recommended.

     The metric collection in load-aware routing has built-in liveness detection: backends from which metrics cannot be collected are automatically removed, providing similar capability to health checks.

   • Select the For Extensible instances checkbox at the bottom of the page. Enable Associate Service Extension and Use Existing Service Extension. Select the ext-load-aware-routing created in Step 3.

2. After Server Group Created, click Add Backend Servers. In the Add Backend Server panel, add the Pod addresses of the deployed inference service (obtainable from ACK cluster list > Details > Network > Services). Click Add IP Address to add multiple entries, then click Next.

3. In the Ports/Weights step, set Port to 8000, keep Weight at the default value, and click OK.

5. Create a listener

1. In the ALB console, click the target instance ID to go to the Instance Details page. On the Listener tab, click Create Listener.

2. In the Configure Listener step, set Listener Protocol to HTTP and Listener Port to 80. In Advanced Settings, set Connection Request Timeout to 3600 seconds. Then click Next.

   This example uses an HTTP listener to simplify the configuration and focus on verifying the scheduling effect of load-aware routing. In a production environment, use an HTTPS listener with certificates configured for transport security.

   LLM inference requests (especially when generating many output tokens) can take a long time per response. If the response time exceeds the listener's default request timeout (60 seconds), requests are terminated prematurely. This example sets the request timeout to 3600 seconds to prevent long responses from being interrupted. Adjust this value based on your actual request duration.

3. In the Select Server Group step, select the server group sgp-load-aware created in Step 4. Then click Next.

4. In the Configuration Review step, confirm the configuration and click Submit.

6. Configure DNS resolution

Point your custom domain to the ALB instance's DNS name via a CNAME record so that clients can access ALB through your custom domain.

This example uses Alibaba Cloud DNS. For domains not registered with Alibaba Cloud, you must first add the domain to the DNS console.

1. In the ALB console, copy the Domain Name of the target instance.

2. Log on to the DNS console. In the Actions column of the target domain, click Settings. On the Settings page, click Add Record.

3. Add a CNAME record with the following information, then click OK.

   • Record Type: Select CNAME.

   • Hostname: Enter a domain prefix such as test. If your root domain is example.com, the domain for accessing ALB would be test.example.com.

   • Query Source and TTL: Keep the default values.

   • Record Value: Enter the DNS name of the ALB instance.

4. In the Change Resource Record Confirmation dialog box that appears, confirm the resolution information and click OK.

7. Verification test

Use the benchmarking tool vllm bench serve to run a stress test against the backends. To avoid interference from public network bandwidth, latency, and jitter on test results, this example runs the benchmark from an internal network environment within the same VPC as ALB: a dedicated benchmark Pod is deployed (with only the vLLM tool installed and the same model mounted, without starting the inference service) as the test client, separate from the tested backends, to avoid consuming inference resources or interfering with metric collection.

By setting a concurrency target far exceeding the service's performance limit, the backend GPUs run under extreme load. This validates whether load-aware routing can optimize request distribution, reduce extreme queuing, and accelerate overall throughput.

vllm bench serve \
  --backend vllm \
  --model /models/DeepSeek-R1-Distill-Qwen-1.5B \
  --served-model-name deepseek-r1-distill-qwen-1.5b \
  --trust-remote-code \
  --dataset-name random \
  --random-prefix-len 1000 \
  --random-input-len 3000 \
  --random-output-len 3000 \
  --random-range-ratio 0.2 \
  --num-prompts 3000 \
  --max-concurrency 600 \
  --host <Service-ClusterIP> \
  --port 8000 \
  --endpoint /v1/completions \
  --save-result \
  2>&1 | tee benchmark_svc.txt

vllm bench serve \
  --backend vllm \
  --model /models/DeepSeek-R1-Distill-Qwen-1.5B \
  --served-model-name deepseek-r1-distill-qwen-1.5b \
  --trust-remote-code \
  --dataset-name random \
  --random-prefix-len 1000 \
  --random-input-len 3000 \
  --random-output-len 3000 \
  --random-range-ratio 0.2 \
  --num-prompts 3000 \
  --max-concurrency 600 \
  --host <ALB-VIP> \
  --port 80 \
  --endpoint /v1/completions \
  --save-result \
  2>&1 | tee benchmark_alb.txt

Billing

• ALB Extensible Edition — Currently in public preview and available for free.

• ACK cluster and GPU instances — The backend inference service runs on GPU nodes in an ACK cluster. Billing follows the corresponding Container Service and ECS instance pricing rules. If created for testing purposes, use pay-as-you-go instances and release them promptly.

• Domain and public DNS resolution fees — In addition to domain fees from your domain provider, configuring public DNS resolution on Alibaba Cloud incurs public authoritative resolution fees.

Limits

• Metric collection requests are sent using HTTP 1.1 over IPv4 addresses. IPv6 backends are not currently supported.

• The metric collection response body for a single backend is limited to 10 KB by default. Content exceeding this limit is discarded, which may cause incomplete metric collection. Load-aware routing only uses three metrics: vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, and vllm:num_requests_running. Place these metrics within the first 10 KB of the metrics output to ensure they are collected properly.

FAQ