DocsICP FilingConsole
官方文档
首页

Use the eGPU Kubernetes component

更新时间：
复制 MD 格式
产品详情
我的收藏

eGPU is a container virtualization solution that enables GPU sharing for cloud-native platforms and large-scale clusters. To share GPU resources in a Kubernetes cluster by using eGPU, install the eGPU device plugin to enable GPU virtualization scheduling.

  • Docker version 19.03.5 or later is recommended. This document uses version 19.03.5.

  • Kubernetes 1.21.2 or later is recommended. This document uses version 1.21.2.

  • Helm 3.0.0 or later must be installed on the Kubernetes master node. This document uses version 3.3.4.

  • Nvidia-docker2 is installed.

  • eGPU is installed.

If you use an ACK-managed cluster, a GPU virtualization scheduling component is installed by default. You can skip the installation steps in this document and proceed to the verification steps. Currently, the ACK scheduling component does not support the compute slicing feature of eGPU.

Installation procedure

  1. Enable the Nvidia runtime.

    1. Modify the /etc/docker/daemon.json file to include the following content.

      {
 "default-runtime": "nvidia",
 "runtimes": {
 "nvidia": {
 "path": "/usr/bin/nvidia-container-runtime",
 "runtimeArgs": []
 }
 }
}

    2. Restart Docker.

      sudo systemctl restart docker

  2. Label the nodes.

    1. Add the egpu label.

      This label allows the device plugin to identify nodes that have eGPU capabilities. On the master node, run the following command. Replace <target_node> with the node name from the kubectl get no command's output, for example, cn-shanghai.gpu1.

      sudo kubectl label node <target_node> egpu=true

    2. Add the node.gpu.placement label.

      This label specifies the scheduling mode for a node. Two scheduling modes are available:

      • binpack: Prioritizes filling a GPU with workloads before assigning workloads to another GPU.

      • spread: Schedules the next workload to the GPU that has the most available resources.

      Note

      If this label is not set, the default mode is binpack.

      sudo kubectl label node <target_node> node.gpu.placement=spread
sudo kubectl label node <target_node> node.gpu.placement=binpack

  3. Install using Helm.

    On the master node, get the installation files. This guide assumes they are in the deployer directory.

    1. Run the following command to install with Helm.

      sudo helm install egpu -n kube-system deployer/

    2. Verify that the installation is successful and the components are running.

      The following is example output from a correctly running cluster with one master node and one GPU node.

      $sudo kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7d75679df-gw7w8 1/1 Running 0 11d
coredns-7d75679df-pl4p9 1/1 Running 0 11d
device-plugin-evict-ds-x5n6p 1/1 Running 0 84m
etcd-harp-03 1/1 Running 0 11d
gpushare-device-plugin-ds-6hxxs 1/1 Running 0 84m
gpushare-installer-scckq 0/2 Completed 0 84m
gpushare-schd-extender-56688dd8d4-kgc6z 1/1 Running 0 84m
kube-apiserver-harp-03 1/1 Running 0 11d
kube-controller-manager-harp-03 1/1 Running 0 11d
kube-flannel-ds-lkh9g 1/1 Running 1 116m
kube-flannel-ds-nj6mk 1/1 Running 0 11d
kube-proxy-f9x7x 1/1 Running 0 116m
kube-proxy-mdp59 1/1 Running 0 11d
kube-scheduler-harp-03 1/1 Running 0 83m

  4. eGPU inspector

    The egpu inspector is a tool installed on the master node that lets you query GPU resource usage across the cluster.

    $sudo kubectl inspect egpu
NAME IPADDRESS GPU0(Allocated/Total) GPU1(Allocated/Total) GPU2(Allocated/Total) GPU3(Allocated/Total) GPU4(Allocated/Total) GPU5(Allocated/Total) GPU6(Allocated/Total) GPU7(Allocated/Total) GPU Memory(GiB)
cn-shanghai.gpu1 100.82.131.102 0/39 0/39 0/39 0/39 0/39 0/39 0/39 0/39 0/312
NAME IPADDRESS GPU0(Allocated/Total) GPU1(Allocated/Total) GPU2(Allocated/Total) GPU3(Allocated/Total) GPU4(Allocated/Total) GPU5(Allocated/Total) GPU6(Allocated/Total) GPU7(Allocated/Total) GPU Compute
cn-shanghai.gpu1 100.82.131.102 0/100 0/100 0/100 0/100 0/100 0/100 0/100 0/100 0/800
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
0/312 (0%)
Allocated/Total GPU Compute In Cluster:
0/800 (0%)

    The output indicates that the cluster has one GPU node with eight GPUs. Each GPU has 39 GiB of GPU memory and 100% GPU compute. Currently, no resources are allocated.

    Note

    This feature is not currently supported in ACK-managed clusters.

  5. Verify the device plugin installation.

    Submit an eGPU job to verify that the device plugin is installed correctly. Assume that your environment has one master node and one GPU node.

    $sudo kubectl get no
NAME STATUS ROLES AGE VERSION
harp-03 Ready control-plane,master 11d v1.21.2
cn-shanghai.gpu1 Ready <none> 127m v1.21.2

    Edit the YAML file to add the following resources and environment variables.

    Note

    In ACK-managed clusters, compute slicing is not supported. You do not need to specify the aliyun.com/gpu-compute field.

    resources:
 limits:
 aliyun.com/gpu-mem: "16"
 aliyun.com/gpu-compute: "40"
 env:
 - name: AMP_VGPU_ENABLE
 value: "1"
 - name: AMP_USE_HOST_DAEMON
 value: "1"
 - name: NVIDIA_DRIVER_CAPABILITIES
 value: all

    The following is a complete YAML sample. Save it as egpu-test.yaml. This job creates eight pods, each requesting 16 GiB of GPU memory and 40% of the GPU compute from a single GPU.

    apiVersion: batch/v1
kind: Job
metadata:
 name: amp-egpu-test
 labels:
 app: amp-egpu-test
spec:
 parallelism: 8
 template: # define the pods specifications
 metadata:
 labels:
 app: test-egpu-001
 spec:
 containers:
 - name: test-egpu-001
 image: registry.cn-beijing.aliyuncs.com/ai-samples/tensorflow:1.5.0-devel-gpu
 command:
 - python
 - tensorflow-sample-code/tfjob/docker/mnist/main.py
 - --max_steps=100000
 - --data_dir=tensorflow-sample-code/data
 resources:
 limits:
 aliyun.com/gpu-mem: "16"
 aliyun.com/gpu-compute: "40"
 env:
 - name: AMP_VGPU_ENABLE
 value: "1"
 - name: AMP_USE_HOST_DAEMON
 value: "1"
 - name: NVIDIA_DRIVER_CAPABILITIES 
 value: all
 restartPolicy: Never

    Apply the YAML file.

    sudo kubectl apply -f egpu-test.yaml

    After a moment, verify the results.

    sudo kubectl get po
NAME READY STATUS RESTARTS AGE
amp-egpu-test-6fntr 1/1 Running 0 66s
amp-egpu-test-6knks 1/1 Running 0 66s
amp-egpu-test-drwgq 1/1 Running 0 66s
amp-egpu-test-fsv48 1/1 Running 0 66s
amp-egpu-test-ldtqw 1/1 Running 0 66s
amp-egpu-test-m6zgz 1/1 Running 0 66s
amp-egpu-test-qhccq 1/1 Running 0 66s
amp-egpu-test-rfb8f 1/1 Running 0 66s
    sudo kubectl inspect egpu
NAME IPADDRESS GPU0(Allocated/Total) GPU1(Allocated/Total) GPU2(Allocated/Total) GPU3(Allocated/Total) GPU4(Allocated/Total) GPU5(Allocated/Total) GPU6(Allocated/Total) GPU7(Allocated/Total) GPU Memory(GiB)
cn-shanghai.gpu1 100.82.131.102 32/39 32/39 32/39 32/39 0/39 0/39 0/39 0/39 128/312
NAME IPADDRESS GPU0(Allocated/Total) GPU1(Allocated/Total) GPU2(Allocated/Total) GPU3(Allocated/Total) GPU4(Allocated/Total) GPU5(Allocated/Total) GPU6(Allocated/Total) GPU7(Allocated/Total) GPU Compute
cn-shanghai.gpu1 100.82.131.102 80/100 80/100 80/100 80/100 0/100 0/100 0/100 0/100 320/800
---------------------------------------------------------------------------------------------------------
Allocated/Total GPU Memory In Cluster:
128/312 (41%)
Allocated/Total GPU Compute In Cluster:
320/800 (40%)

    The output shows that eight pods are scheduled on the cn-shanghai.gpu1 node. Four of the eight GPUs (indexes 0, 1, 2, and 3) are used. Each of these four GPUs has allocated 32 of 39 GiB of GPU memory and 80% of its GPU compute. This result confirms that scheduling is working as expected.

Uninstall

  • Remove the labels from the node.

    sudo kubectl label no <target-node> egpu-
sudo kubectl label no <target-node> node.gpu.placement-

  • Uninstall the device plugin.

    sudo helm uninstall egpu -n kube-system
Previous：eGPU interface usage exampleNext：Maintain clusters