Use NVIDIA MPS (Multi-Process Service) for GPU sharing and GPU memory isolation-Container Service for Kubernetes(ACK)-阿里云帮助中心

Shared GPU scheduling uses NVIDIA MPS (Multi-Process Service) as the underlying GPU isolation module, enabling multiple pods to share one GPU with memory isolation. This topic describes how to enable MPS and integrate it with the shared GPU scheduling component in ACK clusters.

Background

MPI (Message Passing Interface) parallelizes CPU cores across compute tasks, but CUDA-accelerated MPI processes often underutilize the GPU. NVIDIA MPS (Multi-Process Service) lets multiple CUDA applications share a single GPU, improving utilization and throughput in multi-user or many-small-task scenarios.

MPS uses a client-server architecture with binary compatibility, requiring no major changes to existing CUDA applications. It has three components:

Control Daemon Process: Manages the MPS Server lifecycle and client connections.
Client Runtime: Built into the CUDA driver library. Handles communication between applications and the MPS Server, enabling GPU sharing without major code changes.
Server Process: Schedules requests from multiple clients on a single GPU, enabling concurrency.

Usage notes

MPS Clients (GPU applications using MPS) must stay connected to the MPS Control Daemon. If the daemon restarts, clients exit with errors.
The MPS Control Daemon runs as a container, deployed by a DaemonSet (one pod per GPU node):
- Do not delete or restart the MPS Control Daemon pod—GPU applications on the node become unavailable. Check pod status with kubectl get po -l app.aliyun.com/name=mps-control-daemon -A.
- The MPS Control Daemon container requires privileged, hostIPC, and hostPID permissions, which carry security risks. Assess carefully before using this solution.
- The MPS Control Daemon pod uses priorityClassName: system-node-critical to avoid termination when node resources run low. During deployment, it may preempt lower-priority pods if resources are insufficient. Ensure nodes have sufficient CPU and memory before deploying.
For GPU nodes that are managed in Container Service for Kubernetes (ACK) clusters, you need to pay attention to the following items when you request GPU resources for applications and use GPU resources.
- Do not run GPU-heavy applications directly on nodes.
- Do not use tools, such as Docker, Podman, or nerdctl, to create containers and request GPU resources for the containers. For example, do not run the docker run --gpus all or docker run -e NVIDIA_VISIBLE_DEVICES=all command and run GPU-heavy applications.
- Do not add the NVIDIA_VISIBLE_DEVICES=all or NVIDIA_VISIBLE_DEVICES=<GPU ID> environment variable to the env section in the pod YAML file. Do not use the NVIDIA_VISIBLE_DEVICES environment variable to request GPU resources for pods and run GPU-heavy applications.
- Do not set NVIDIA_VISIBLE_DEVICES=all and run GPU-heavy applications when you build container images if the NVIDIA_VISIBLE_DEVICES environment variable is not specified in the pod YAML file.
- Do not add privileged: true to the securityContext section in the pod YAML file and run GPU-heavy applications.
The following potential risks may exist when you use the preceding methods to request GPU resources for your application:
- If you use one of the preceding methods to request GPU resources on a node but do not specify the details in the device resource ledger of the scheduler, the actual GPU resource allocation information may be different from that in the device resource ledger of the scheduler. In this scenario, the scheduler can still schedule certain pods that request the GPU resources to the node. As a result, your applications may compete for resources provided by the same GPU, such as requesting resources from the same GPU, and some applications may fail to start up due to insufficient GPU resources.
- Using the preceding methods may also cause other unknown issues, such as the issues reported by the NVIDIA community.

Prerequisites

An ACK managed cluster Pro edition running version 1.20 or later. If your version is older, upgrade your cluster.

Procedure

Step 1: Install the MPS Control Daemon component

Log on to the ACK console. In the left navigation pane, click Marketplace > Marketplace.
In the Marketplace, search for ack-mps-control and click it.
On the ack-mps-control page, click Deploy. Select the target Cluster and click Next.
On the Create page, select the Chart Version and click OK.
Important
- Uninstalling or upgrading ack-mps-control causes running GPU applications on the node to exit with errors. Perform during off-peak hours.
- The upgrade strategy is OnDelete—pods do not restart automatically. After upgrading, manually delete old ack-mps-control DaemonSet pods. See How do I upgrade the MPS Control Daemon component?

Step 2: Install the shared GPU component

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Cloud-native AI Suite.
On the Cloud-native AI Suite page, click Deploy.
On the Deploy Now page for the Cloud-native AI Suite, select Scheduling Policy Extension (Batch Task Scheduling, GPU Sharing, Topology-aware GPU Scheduling).
On the Cloud-native AI Suite page, click Deploy Cloud-native AI Suite.

After installation, find ack-ai-installer in the component list on the Cloud-native AI Suite page.

Step 3: Enable GPU sharing scheduling and GPU memory isolation

On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.
On the Node Pools page, click Create Node Pool.

On the Create Node Pool page, configure the node pool settings and click Confirm.

See Create and manage node pools for other settings.

Setting	Description
Desired number of nodes	The initial number of nodes in the node pool. Note After creating the node pool, add GPU nodes to it. When doing so, ensure you specify a GPU cloud server as the instance type. For more information, see Add existing nodes to a node pool or Create and manage a node pool.
Node labels	Click for Node Label, set Key to `ack.node.gpu.schedule` and Value to `mps`. Important Label each GPU node with `ack.node.gpu.schedule=mps` to deploy the MPS Control Daemon Pod. If the shared GPU scheduling component is installed, this label also enables shared GPU scheduling and MPS isolation on the node. After adding the shared GPU scheduling label, do not use the `kubectl label nodes` command to change the GPU scheduling label or use the label management feature on the Nodes page to change the node label. This can cause potential issues. For more information, see Enable scheduling. The recommended method is described in Enable scheduling.

Step 4: Deploy a sample application

Create a sample application with the following YAML file.

apiVersion: batch/v1
kind: Job
metadata:
  name: mps-sample
spec:
  parallelism: 1
  template:
    metadata:
      labels:
        app: mps-sample
    spec:
      hostIPC: true  # Required. Otherwise, the pod fails to start.
      hostPID: true  # Optional. Added here only to help you see the effect of MPS.
      nodeSelector:
       kubernetes.io/hostname: <NODE_NAME> # Replace <NODE_NAME> with the hostname of a GPU node that has the label ack.node.gpu.schedule=mps. For example: cn-shanghai.192.0.2.109.
      containers:
      - name: mps-sample
        image: registry.cn-hangzhou.aliyuncs.com/ai-samples/gpushare-sample:tensorflow-1.5
        command:
        - python
        - tensorflow-sample-code/tfjob/docker/mnist/main.py
        - --max_steps=100000
        - --data_dir=tensorflow-sample-code/data
        resources:
          limits: 
            aliyun.com/gpu-mem: 7  # Request 7 GiB of GPU memory for this pod.
        workingDir: /root
      restartPolicy: Never

Important

GPU application pods on MPS-enabled nodes must set hostIPC: true, or they fail to start.

After the pod reaches Running state, check whether MPS is active:

kubectl exec -ti mps-sample-xxxxx --  nvidia-smi

Expected output:

Tue May 27 05:32:12 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.07             Driver Version: 535.161.07   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla ****-****-****           On  | 00000000:00:09.0 Off |                    0 |
| N/A   33C    P0              55W / 300W |    345MiB / 16384MiB |      0%   E. Process |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A     14732      C   nvidia-cuda-mps-server                       30MiB |
|    0   N/A  N/A    110312    M+C   python                                      312MiB |
+---------------------------------------------------------------------------------------+

The nvidia-smi output lists the mps-server process (PID 14732) and a Python process (PID 110312) running under MPS, confirming MPS is active.

FAQ

How do I upgrade the MPS Control Daemon component?

ack-mps-control v0.2.0 upgrade requires ack-ai-installer >= 1.13.1. Follow this order:

On the Cloud-native AI Suite page, upgrade the Helm version of ack-ai-installer.
In Applications > Helm, select the kube-system namespace and upgrade ack-mps-control.

The upgrade strategy is OnDelete—pods do not restart automatically. After upgrading, manually delete old ack-mps-control DaemonSet pods.
For each node, perform Drain, mark it as Unschedulable, and delete the ack-mps-control pod.
1. Mark the node as Unschedulable and click Drain.
2. Delete the ack-mps-control pod on that node.
3. Confirm that the new pod runs normally.
Delete the ack-ai-installer pod.

After upgrading ack-mps-control and confirming pods are updated, delete the ack-ai-installer pod. It rebuilds automatically.
Mark the node as schedulable.

After confirming both ack-mps-control and ack-ai-installer pods run normally on the node, mark it as schedulable.