DocsICP FilingConsole
官方文档
首页

Enable inventory-aware elastic scheduling for multi-cluster fleets

更新时间：
复制 MD 格式
产品详情
我的收藏

In multi-region fleet deployments, schedule GPU workloads using real-time ECS inventory and instant node elasticity to maximize placement success and avoid idle GPU nodes.

Important

This feature is in invitational preview. To request access, submit a ticket.

How it works

In GPU inference scenarios, GPU supply varies by region and pre-provisioned GPU nodes incur high idle costs. Inventory-aware scheduling with instant node elasticity addresses both challenges. Three components enable inventory-aware elastic scheduling:

Component

Role

Fleet scheduler

Detects shortfalls in child clusters, queries inventory via ACK GOATScaler, and distributes replicas by available capacity—including inventory not yet provisioned as nodes.

ACK GOATScaler

Runs in each child cluster, checks real-time ECS inventory, and returns available instance counts to the fleet scheduler.

Child cluster node pools

Uses instant node elasticity with zero desired nodes. Nodes scale out when the scheduler assigns workloads and scale in when workloads are removed.

When you deploy an application to a fleet and no child cluster has enough running resources:

  1. The scheduler detects insufficient resources in child clusters and cannot schedule the workload.

  2. The scheduler triggers ACK GOATScaler in each child cluster to query inventory.

  3. The scheduler redistributes the application to the cluster with available inventory.

  4. The selected cluster scales out nodes and runs the application.

image

Prerequisites

You need:

Important

If node autoscaling is enabled on a child cluster, switch to instant node elasticity. See Enable instant node elasticity.

GPU instance specifications and cost estimation

Model parameters consume most GPU memory during inference. Estimate required GPU memory with:

GPU memory = Number of parameters × Bytes per parameter

Example: 7B model, FP16 precision

Factor

Value

Parameters

7 × 10⁹

Bytes per parameter (FP16)

2 bytes

Model memory

7 × 10⁹ × 2 bytes ≈ 13.04 GiB

Account for KV cache and computation buffers beyond model loading. For a 7B FP16 model, use a GPU instance with at least 24 GiB GPU memory, such as ecs.gn7i-c8g1.2xlarge or ecs.gn7i-c16g1.4xlarge.

For full instance type details and pricing, see GPU-accelerated compute optimized instance family and Elastic GPU Service billing.

Prepare model files and storage

Download the Qwen3-8B model and create OSS PersistentVolumes in each child cluster.

1. Download the model

Note

Install Git Large File Storage (LFS) if needed: run yum install git-lfs or apt-get install git-lfs. See Install Git Large File Storage.

git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/Qwen/Qwen3-8B.git
cd Qwen3-8B
git lfs pull

2. Upload the model to OSS

Note

See Install ossutil.

ossutil mkdir oss://<your-bucket-name>/models/Qwen3-8B
ossutil cp -r ./Qwen3-8B oss://<your-bucket-name>/models/Qwen3-8B

3. Create a PersistentVolume and PersistentVolumeClaim in each child cluster

See Use ossfs 1.0 static persistent volume.

apiVersion: v1
kind: Secret
metadata:
  name: oss-secret
stringData:
  akId: <your-oss-ak>      # AccessKey ID for accessing OSS
  akSecret: <your-oss-sk>  # AccessKey Secret for accessing OSS
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: qwen3-8b
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 20Gi
  selector:
    matchLabels:
      alicloud-pvname: qwen3-8b
  storageClassName: oss
  volumeMode: Filesystem
  volumeName: qwen3-8b
---
apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    alicloud-pvname: qwen3-8b
  name: qwen3-8b
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 20Gi
  csi:
    driver: ossplugin.csi.alibabacloud.com
    nodePublishSecretRef:
      name: oss-secret
      namespace: default
    volumeAttributes:
      bucket: <your-bucket-name>       # Bucket name
      otherOpts: '-o allow_other -o umask=000'
      path: <your-model-path>          # Example: /models/Qwen3-8B/
      url: <your-bucket-endpoint>      # Example: oss-cn-hangzhou-internal.aliyuncs.com
    volumeHandle: qwen3-8b
  persistentVolumeReclaimPolicy: Retain
  storageClassName: oss
  volumeMode: Filesystem

Step 1: Configure node pools for child clusters

In each child cluster, create or edit a GPU node pool with these settings:

Setting

Value

Instance type

ecs.gn7i-c8g1.2xlarge

Scaling Mode

Auto

Desired nodes

0

Zero desired nodes eliminate idle GPU costs. The pool scales out only when the fleet scheduler assigns workloads.

See Create and manage node pools.

Note

Shorten the scale-in trigger delay in node pool settings to reduce wait time in Step 3.

Step 2: Create an application and distribution policy in the fleet cluster

Use the fleet cluster's kubeconfig for all resources in this step.

1. Create deploy.yaml

The Deployment runs 4 Qwen3-8B replicas with vLLM, each pod requesting 1 GPU.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: qwen3-8b
  name: qwen3-8b
  namespace: default
spec:
  replicas: 4
  selector:
    matchLabels:
      app: qwen3-8b
  template:
    metadata:
      labels:
        app: qwen3-8b
    spec:
      volumes:
        - name: qwen3-8b
          persistentVolumeClaim:
            claimName: qwen3-8b
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 20Gi
      containers:
      - command:
        - sh
        - -c
        - vllm serve /models/qwen3-8b --port 8000 --trust-remote-code --served-model-name qwen3-8b --tensor-parallel=1 --max-model-len 8192 --gpu-memory-utilization 0.95 --enforce-eager
        image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm-openai:v0.9.1
        name: vllm
        ports:
        - containerPort: 8000
        readinessProbe:
          tcpSocket:
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 30
        resources:
          limits:
            nvidia.com/gpu: "1"
        volumeMounts:
          - mountPath: /models/qwen3-8b
            name: qwen3-8b
          - mountPath: /dev/shm
            name: dshm

2. Create PropagationPolicy.yaml

The PropagationPolicy distributes the Deployment across two child clusters.

Key fields

Field

Value

Description

autoScaling.ecsProvision

true

Enables inventory-aware elastic scheduling and queries real-time ECS inventory when placing workloads.

replicaSchedulingType

Divided

Distributes replicas across clusters. Use Duplicated to deploy a full copy of the Deployment to each cluster instead.

dynamicWeight: AvailableReplicas

—

Allocates replicas proportionally to each cluster's schedulable capacity—including ECS inventory. Clusters with available inventory get more replicas, even without running GPU nodes.

 
apiVersion: policy.one.alibabacloud.com/v1alpha1
kind: PropagationPolicy
metadata:
  name: demo-policy
spec:
  # Enables inventory-aware elastic scheduling
  autoScaling:
    ecsProvision: true
  preserveResourcesOnDeletion: false
  conflictResolution: Overwrite
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    name: qwen3-8b
    namespace: default
  placement:
    replicaScheduling:
      replicaSchedulingType: Divided
      weightPreference:
        dynamicWeight: AvailableReplicas
    clusterAffinity:
      clusterNames:
      - ${cluster1-id}  # Replace with your actual child cluster ID
      - ${cluster2-id}  # Replace with your actual child cluster ID

3. Apply the manifests

kubectl apply -f deploy.yaml
kubectl apply -f PropagationPolicy.yaml

GPU node pools in both child clusters begin scaling out shortly after.

Step 3: Verify elastic scaling

Check workload scheduling status

kubectl get resourcebinding

Example output:

NAME                  SCHEDULED   FULLYAPPLIED   OVERRIDDEN   ALLAVAILABLE   AGE
qwen3-8b-deployment   True        True           True         False          7m47s

Field

Value

Meaning

SCHEDULED

True

The fleet scheduler placed the workload across child clusters.

ALLAVAILABLE

False

Nodes are still scaling out—a normal intermediate state, not an error. Changes to True once all pods are running.

Check pod distribution across clusters

When pods reach the Running state, run:

kubectl amc get deploy qwen3-8b -M

Example output:

NAME       CLUSTER           READY   UP-TO-DATE   AVAILABLE   AGE     ADOPTION
qwen3-8b   cxxxxxxxxxxxxxx   2/2     2            2           3m22s   Y
qwen3-8b   cxxxxxxxxxxxxxx   2/2     2            2           3m22s   Y

Field

Value

Meaning

READY

2/2

All replicas in the cluster are running.

ADOPTION

Y

The cluster owns the workload.

All 4 replicas run across both clusters, even though both had zero GPU nodes before deployment.

Verify scale-in behavior

Scale the Deployment to 2 replicas and re-apply:

kubectl apply -f deploy.yaml
Note

Or delete the workload to simulate zero replicas.

After ten minutes, GPU node pools in each child cluster scale in to one node. If you deleted the workload, nodes scale in to zero.

This confirms the full workflow: the fleet schedules workloads from real-time inventory, scales out GPU nodes to run them, and scales back in when workloads are removed—eliminating idle GPU costs.

Next steps

Previous：Application distribution status and FAQNext：Migrate a single-cluster application to a fleet and distribute it to multiple clusters