Use GPU sharing on GPU-HPN nodes

Pod states and Quality of Service

Similar to OS process management, the GPU sharing module assigns each pod one of three states:

• Hibernation: The pod has no GPU demand (initial state when a pod starts).

• Ready: The pod is waiting for GPU resources to be allocated.

• Running: The pod is actively using GPU resources.

When multiple pods compete for GPU resources simultaneously, the GPU sharing module applies Quality of Service (QoS) policies to manage allocation fairly.

Queuing policy (share-pool model only)

Pods in the ready state are queued using First In, First Out (FIFO). The pod that entered the ready state first receives resources first. If resources are insufficient, the preemption policy is triggered.

Preemption policy (share-pool model only)

When a pod in the ready queue cannot get resources, the GPU sharing module attempts to preempt a running pod using the following criteria:

If no running pod meets the filter condition, the queued pod continues to wait.

Choose a sharing model

ACS supports two sharing models. The main difference is whether a pod is assigned a fixed GPU device or can use any available GPU on the node:

• Use share-pool when workloads are bursty or intermittent (such as Notebook sessions). Pods share resources from a common GPU pool, with QoS mechanisms (queuing and preemption) managing contention.

• Use static when workloads need guaranteed, uninterrupted GPU access without queuing. Each pod is fixed to a specific GPU device.

Example: off-peak resource sharing in Notebook scenarios

In Notebook development, workloads typically do not hold GPU resources continuously. With the share-pool model, pods use GPU resources only when they need them, and the QoS mechanism manages access when multiple pods request resources simultaneously.

Consider four pods on a node with two GPUs:

• Pods A and B: requests=0.5, limits=0.5

• Pods C and D: requests=0.5, limits=1

Based on requests, all four pods fit on the node.

Time T1: Pods A and C are running. Pods B and D are in the ready queue. The GPU sharing module tries to allocate resources to Pod D (first in queue). GPU 0 has 0.5 GPU idle, which satisfies Pod D's requests=0.5, but Pod D's limit=1 would cause resource competition with Pod A on the same GPU. So Pod D stays in the queue.

Time T2 – Phase 1: Pod C finishes and enters hibernation. GPU 1 becomes free and its resources are allocated to Pod D.

Time T2 – Phase 2: Pod B is allocated resources on GPU 0. Because Pod B's limit=0.5, it can share GPU 0 with Pod A without resource competition.

Prerequisites

Before you begin, ensure that you have:

• An ACS cluster with GPU-HPN nodes.

• kubectl configured to connect to the cluster.

• Deleted any pods on the target node that request exclusive GPU resources. Pods that use only CPU and memory do not need to be deleted.

Step 1: Label the GPU-HPN node

List the GPU-HPN nodes in the cluster:

kubectl get node -l alibabacloud.com/node-type=reserved

Expected output:

NAME                     STATUS   ROLES   AGE   VERSION
cn-wulanchabu-c.cr-xxx   Ready    agent   59d   v1.28.3-aliyun

Add the alibabacloud.com/gpu-share-policy=share-pool label to enable GPU sharing:

kubectl label node cn-wulanchabu-c.cr-xxx alibabacloud.com/gpu-share-policy=share-pool

Step 2: Verify the node status

After applying the label, check that the feature is active on the node:

kubectl get node cn-wulanchabu-c.cr-xxx -o yaml

Expected output (truncated):

# The actual output may vary.
apiVersion: v1
kind: Node
spec:
  # ...
status:
  allocatable:
    # GPU shared resource description
    alibabacloud.com/gpu-core.percentage: "1600"
    alibabacloud.com/gpu-memory.percentage: "1600"
    # CPU, memory, and storage reserved for the GPU sharing module
    cpu: "144"
    memory: 1640Gi
    nvidia.com/gpu: "16"
    ephemeral-storage: 4608Gi
  capacity:
    # GPU shared resource description
    alibabacloud.com/gpu-core.percentage: "1600"
    alibabacloud.com/gpu-memory.percentage: "1600"
    cpu: "176"
    memory: 1800Gi
    nvidia.com/gpu: "16"
    ephemeral-storage: 6Ti
  conditions:
  # Indicates whether the GPU share policy configuration is valid
  - lastHeartbeatTime: "2025-01-07T04:13:04Z"
    lastTransitionTime: "2025-01-07T04:13:04Z"
    message: gpu share policy is valid.
    reason: Valied
    status: "True"
    type: GPUSharePolicyValid
  # Indicates the GPU share policy in effect on this node
  - lastHeartbeatTime: "2025-01-07T04:13:04Z"
    lastTransitionTime: "2025-01-07T04:13:04Z"
    message: gpu share policy is share-pool.
    reason: share-pool
    status: "True"
    type: GPUSharePolicy

Confirm the following in the output to verify that the feature is active:

• allocatable and capacity include alibabacloud.com/gpu-core.percentage and alibabacloud.com/gpu-memory.percentage.

• The GPUSharePolicyValid condition has status: "True".

• The GPUSharePolicy condition has reason: share-pool.

If the node resources do not update as described, the configuration failed. Check the GPUSharePolicyValid condition's reason and message fields for details. See Node conditions for reason values.

Step 3: Deploy a pod with shared GPU resources

Create a file named gpu-share-demo.yaml. Set the GPU sharing model to share-pool, matching the node configuration:

apiVersion: v1
kind: Pod
metadata:
  labels:
    alibabacloud.com/compute-class: "gpu-hpn"
    # Set the GPU sharing model to share-pool, matching the node configuration
    alibabacloud.com/gpu-share-policy: "share-pool"
  name: gpu-share-demo
  namespace: default
spec:
  containers:
  - name: demo
    image: registry-cn-wulanchabu-vpc.ack.aliyuncs.com/acs/stress:v1.0.4
    args:
      - '1000h'
    command:
      - sleep
    resources:
      limits:
        cpu: '5'
        memory: 50Gi
        alibabacloud.com/gpu-core.percentage: 100   # Upper limit of computing power usage
        alibabacloud.com/gpu-memory.percentage: 100  # Upper limit of GPU memory usage; exceeding this causes a CUDA OOM error
      requests:
        cpu: '5'
        memory: 50Gi
        alibabacloud.com/gpu-core.percentage: 10    # Controls how many pods can be scheduled on the node
        alibabacloud.com/gpu-memory.percentage: 10   # Controls how many pods can be scheduled on the node

Deploy the pod:

kubectl apply -f gpu-share-demo.yaml

Step 4: Check GPU resource usage

Log in to the container to verify GPU resource usage:

kubectl exec -it gpu-share-demo -- /bin/bash

Inside the container, use nvidia-smi to view GPU resource allocation and usage. The command to use depends on your GPU card type—nvidia-smi applies to NVIDIA GPU devices. For other card types, submit a ticketsubmit a ticket for assistance.

Step 5 (optional): Disable GPU sharing on the node

1. Delete the pod:

   kubectl delete pod gpu-share-demo

2. Set the GPU sharing policy to none:

   kubectl label node cn-wulanchabu-c.cr-xxx alibabacloud.com/gpu-share-policy=none

3. Verify the node status:

   • allocatable and capacity no longer include alibabacloud.com/gpu-core.percentage or alibabacloud.com/gpu-memory.percentage.

   • The GPUSharePolicy condition has status: "False" and reason: none.

   • CPU and memory in allocatable are restored to their original values.

   kubectl get node cn-wulanchabu-c.cr-xxx -o yaml

   Expected output (truncated):

   apiVersion: v1
   kind: Node
   spec:
     # ...
   status:
     allocatable:
       # Reserved CPU and memory are restored after the feature is disabled
       cpu: "176"
       memory: 1800Gi
       nvidia.com/gpu: "16"
       ephemeral-storage: 4608Gi
     capacity:
       cpu: "176"
       memory: 1800Gi
       nvidia.com/gpu: "16"
       ephemeral-storage: 6Ti
     conditions:
     - lastHeartbeatTime: "2025-01-07T04:13:04Z"
       lastTransitionTime: "2025-01-07T04:13:04Z"
       message: gpu share policy config is valid.
       reason: Valid
       status: "True"
       type: GPUSharePolicyValid
     - lastHeartbeatTime: "2025-01-07T04:13:04Z"
       lastTransitionTime: "2025-01-07T04:13:04Z"
       message: gpu share policy is none.
       reason: none
       status: "False"
       type: GPUSharePolicy

   Confirm the following in the output:

Enablement label

Set the alibabacloud.com/gpu-share-policy label on a node to enable or disable GPU sharing.

apiVersion: v1
kind: Node
metadata:
  labels:
    alibabacloud.com/gpu-share-policy: share-pool  # or: static, none

QoS configuration

Configure Quality of Service (QoS) parameters for GPU sharing using the alibabacloud.com/gpu-share-qos-config node annotation. These parameters apply only to the share-pool model.

apiVersion: v1
kind: Node
metadata:
  annotations:
    alibabacloud.com/gpu-share-qos-config: '{"preemptEnabled": true, "podMaxDurationMinutes": 120, "reservedEphemeralStorage": "1.5Ti"}'

Shared resource fields

When GPU sharing is enabled, the following fields are added to the node's allocatable and capacity. They are removed when the feature is disabled.

Node conditions

The node conditions field reports two GPU sharing condition types.

GPUSharePolicyValid — whether the GPU sharing configuration is valid:

GPUSharePolicy — the currently active GPU sharing policy:

Compute class

GPU sharing policy

Resource requests

Specify GPU shared resources in the container's resources field using percentages of a single GPU's computing power and memory.

Both alibabacloud.com/gpu-core.percentage and alibabacloud.com/gpu-memory.percentage must be specified in both requests and limits.

The number of pods that can be scheduled on a node is also constrained by CPU, memory, and the node's maximum pod count.

What happens to a pod waiting in the ready queue?

The pod periodically logs its waiting status:

You have been waiting for ${1} seconds. Approximate position: ${2}

${1} is the number of seconds the pod has been waiting. ${2} is its current position in the ready queue.

What monitoring metrics are available for GPU sharing pods?

The following metrics are available for share-pool pods:

How do GPU utilization metrics differ for shared GPU pods?

GPU utilization metrics work the same way as for exclusive GPU pods, with a few differences for shared GPU pods:

• ACS pod monitoring: GPU computing power utilization and GPU memory usage are absolute values based on the entire GPU card—the same as in exclusive GPU scenarios.

• In-container view (e.g., nvidia-smi): GPU memory usage is an absolute value, but computing power utilization is a relative value where the denominator is the pod's limit.

• Device IDs: The device ID in metrics corresponds to the actual ID on the node and does not always start from 0.

• share-pool model: The device number in metrics may change because the pod can use different GPU devices from the pool over time.

How do I prevent scheduling conflicts when GPU sharing is enabled on only some nodes?

The default ACS scheduler automatically matches pod and node types, avoiding conflicts.

If you use a custom scheduler, an exclusive GPU pod might be scheduled onto a GPU sharing node because the node exposes both nvidia.com/gpu and GPU shared resources in its capacity. Use one of these approaches:

• Scheduler plugin: Write a plugin that reads ACS node labels and Condition fields to filter out nodes with a mismatched GPU sharing policy. See Scheduling Framework.

• Labels or taints: Add a label or taint to GPU sharing nodes, then configure affinity or toleration policies on your pods.

What information is available when a GPU sharing pod is preempted?

For share-pool pods, preemption generates both an Event and a Condition on the pod.

Events:

# This pod's GPU resources were preempted by <new-pod-name>
Warning  GPUSharePreempted  5m15s  gpushare   GPU is preempted by <new-pod-name>.
# This pod preempted GPU resources from <old-pod-name>
Warning  GPUSharePreempt    3m47s  gpushare   GPU is preempted from <old-pod-name>.

Condition:

- type: Interruption.GPUShareReclaim   # Condition type for GPU sharing preemption events
  status: "True"                        # True: a preemption or preemption-by action occurred
  reason: GPUSharePreempt               # GPUSharePreempt: this pod preempted another pod; GPUSharePreempted: this pod was preempted
  message: GPU is preempted from <old-pod-name>.
  lastTransitionTime: "2025-04-22T08:12:09Z"
  lastProbeTime: "2025-04-22T08:12:09Z"

How do I maximize pod density in a Notebook scenario?

For GPU sharing pods, you can also set CPU and memory requests lower than limits to increase pod density on a node. When the total limits across pods on a node exceeds the node's allocatable resources, pods compete for CPU and memory.

• CPU: Competition shows up as CPU Steal Time in the pod's metrics.

• Memory: Competition can trigger a node-level out-of-memory (OOM) error, causing some pods to be killed.

Plan pod priorities and resource specifications based on each application's characteristics. For node-level resource utilization data, see ACS GPU-HPN node-level monitoring metrics.