Configure and manage the ack-nvidia-device-plugin add-on

Check the NVIDIA device plugin version

kubectl get pods -n kube-system -l component=nvidia-device-plugin \
  -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\t"}{.spec.nodeName}{"\n"}{end}' \
  | awk -F'[:/]' '{split($NF, a, "-"); print a[1] "\t" $0}' \
  | sort -k1,1V \
  | cut -f2- \
  | awk -F'\t' '{split($1, img, ":"); print img[NF] "\t" $2}'

1. Upgrade the ack-nvidia-device-plugin add-on.

   Version 1.32 and later

   1. Log on to the ACK console. In the left navigation pane, click Clusters.

   2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Add-ons.

   3. On the Add-ons page, find the ack-nvidia-device-plugin card and click Upgrade.

   4. In the dialog box that appears, click OK.

   Versions 1.20 to 1.31

   1. On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Nodes.

   2. Select the GPU-accelerated nodes that you want to manage, click Batch Operations at the bottom of the node list, select Run Shell Scripts in the Batch Operations dialog box, and then click OK.

      Important

      We recommend that you first upgrade the GPU device plugin on a small number of GPU-accelerated nodes. After you verify that the plugin works as expected, perform the upgrade on the remaining nodes.

   3. You are redirected to the CloudOps Orchestration Service (OOS) console. Set Execution Mode to Paused Upon Failure and click Next: Parameter Settings.

   4. On the parameter settings page, select Run Shell Script and paste the following sample script.

      Note

      • Set the RUN_PKG_VERSION parameter in the script to the major version number of the cluster (for example, 1.30). Do not enter the minor version number (for example, 1.30.1). Otherwise, the script will report an error.

      • In the script, change the REGION_ID parameter to the region ID of the current cluster, for example, cn-beijing.

      #!/bin/bash
      set -xe
      
      # Set the Kubernetes major version.
      RUN_PKG_VERSION=1.30
      # Set the region ID.
      REGION_ID=cn-beijing
      
      function update_device_plugin() {
      	base_dir=/tmp/update_device_plugin
      	rm -rf $base_dir
      	mkdir -p $base_dir
      	cd $base_dir
      	region_id=$REGION_ID
      	PKG_URL=https://aliacs-k8s-${region_id}.oss-${region_id}-internal.aliyuncs.com/public/pkg/run/run-${RUN_PKG_VERSION}.tar.gz
      	curl -sSL --retry 3 --retry-delay 2 -o run.tar.gz $PKG_URL
      	tar -xf run.tar.gz
      
      	local dir=pkg/run/$RUN_PKG_VERSION/module
      	# Replace the image registry address.
      	sed -i "s@registry.cn-hangzhou.aliyuncs.com/acs@registry-${region_id}-vpc.ack.aliyuncs.com/acs@g" $dir/nvidia-device-plugin.yml
      	mkdir -p /etc/kubernetes/device-plugin-backup
      	mkdir -p /etc/kubernetes/manifests
      	# Back up the old configuration file.
      	mv  /etc/kubernetes/manifests/nvidia-device-plugin.yml /etc/kubernetes/device-plugin-backup/nvidia-device-plugin.yml.$(date +%s)
      	sleep 5
      	# Copy the new configuration file.
      	cp -a $dir/nvidia-device-plugin.yml /etc/kubernetes/manifests
      	echo "succeeded to update device plugin"
      }
      
      # Check if the nvidia-device-plugin.yml file exists.
      if [ -f /etc/kubernetes/manifests/nvidia-device-plugin.yml ]; then
      	update_device_plugin
      else
      	echo "skip to update device plugin"
      fi

   5. Click Next: Confirm. After you confirm the information, click Create.

      After the task is created, you are automatically redirected to the Task Execution Management page, where you can view the task's execution status. The update is successful if the Output is succeeded to update device plugin.

2. Check whether the add-on runs as expected.

   Run the following commands to check if the GPU device plugin is working correctly on the GPU-accelerated node.

   1. Use kubectl to connect to the cluster in Workbench or CloudShell.

   2. Run the following command to check if the NVIDIA Device Plugin has restarted:

      kubectl get po -n kube-system -l component=nvidia-device-plugin 

      The AGE column in the sample output shows whether the pod has restarted.

      NAME                             READY   STATUS    RESTARTS      AGE
      nvidia-device-plugin-xxxx        1/1     Running   1             1m

   3. After all pods restart, run the following script to check if the nodes are reporting GPU resources:

      #!/bin/bash
      
      # Get all NVIDIA Device Plugin pods and their corresponding nodes.
      PODS=$(kubectl get po -n kube-system -l component=nvidia-device-plugin -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}')
      
      # Iterate through each pod's node.
      echo "$PODS" | while IFS=$'\t' read -r pod_name node_name; do
          # Get the allocatable nvidia.com/gpu resource value for the node.
          gpu_allocatable=$(kubectl get node "$node_name" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}' 2>/dev/null)
      
          # Check if the resource value is 0.
          if [ "$gpu_allocatable" == "0" ]; then
              echo "Error: node=$node_name, pod=$pod_name, resource(nvidia.com/gpu) is 0"
          fi
      done

      If a node reports zero resources, see Restart the NVIDIA Device Plugin.

Requirements

Starting from v0.5.0, ack-nvidia-device-plugin supports mounting GDR devices into GPU containers. This feature requires the following:

• The cluster version is 1.32 or later.

• The node is a Lingjun node.

• GDR software is installed on the node.

  You can log in to the node and run the lsmod | grep gdrdrv command to check whether the GDR kernel module is loaded. The sample output is as follows:

  gdrdrv                196608  0
  nvidia              14680064  88 nvidia_uvm,nvidia_peermem,gdrdrv,nvidia_modeset

• NVIDIA Container Toolkit 1.15.0 or later is installed on the node.

  You can check the version by logging in to the node and running nvidia-container-cli --version. The following provides an example of the output:

  cli-version: 1.17.8
  lib-version: 1.17.8

Disable the native GPU isolation feature

Background

When a GPU on a node becomes faulty, ACK automatically isolates the faulty GPU by using the NVIDIA Device Plugin to prevent the scheduler from assigning tasks to it. However, this automatic isolation does not perform an automatic repair. You still need to manually restart or repair the node. We recommend that you configure alerts for GPU exceptions to handle them promptly. For more information about the native GPU isolation feature of the NVIDIA Device Plugin, see k8s-device-plugin.

• After isolation, if the node has insufficient GPUs for a task's requirements (for example, a task needs eight cards but only seven are available), the scheduler cannot place the task on that node, leaving GPU resources idle.

• When the GPU status returns to normal, the system automatically removes the isolation from the device.

• If you need to disable automatic isolation so that the node still reports resources for faulty GPUs, refer to the following solution.

Solution

1. Confirm whether the native GPU isolation feature is enabled for the add-on.

   If it is not enabled, you do not need to perform the subsequent steps to disable it.

   Version 1.32 and later

   Log on to the ACK console and check the ack-nvidia-device-plugin version on the Add-ons page of the target cluster. If the version is v0.1.0, the GPU isolation feature is enabled.

   Versions 1.20 to 1.31

   1. Use kubectl to connect to the cluster in Workbench or CloudShell.

   2. Check if the NVIDIA Device Plugin has GPU isolation enabled.

      check_gpu_isolation_enabled() {
          echo "Checking nodes with NVIDIA Device Plugin deployed as static pods that have GPU isolation enabled (i.e., DP_DISABLE_HEALTHCHECKS is not set to 'all')..."
      
          # Get pod names with the expected label.
          pods=$(kubectl get pods -n kube-system -l component=nvidia-device-plugin --no-headers -o custom-columns=":metadata.name" 2>/dev/null)
      
          if [ -z "$pods" ]; then
              echo "No pods found with label 'component=nvidia-device-plugin'."
              return 0
          fi
      
          found=0
          while IFS= read -r pod; do
              [ -z "$pod" ] && continue
      
              # Check if it's a static pod: annotation kubernetes.io/config.source must be "file".
              config_source=$(kubectl get pod "$pod" -n kube-system -o jsonpath='{.metadata.annotations.kubernetes\.io/config\.source}' 2>/dev/null)
      
              if [ "$config_source" != "file" ]; then
                  # Not a static pod (e.g., managed by a DaemonSet), skip.
                  continue
              fi
      
              # Get node name.
              node=$(kubectl get pod "$pod" -n kube-system -o jsonpath='{.spec.nodeName}' 2>/dev/null)
              [ -z "$node" ] && continue
      
              # Check if DP_DISABLE_HEALTHCHECKS is set to "all".
              disabled_value=$(kubectl get pod "$pod" -n kube-system -o jsonpath='{range .spec.containers[*]}{range .env[?(@.name=="DP_DISABLE_HEALTHCHECKS")]}{.value}{"\n"}{end}{end}' 2>/dev/null | head -n1)
      
              # If not set or not equal to "all", health checks are active, which means isolation is enabled.
              if [ -z "$disabled_value" ] || [ "$disabled_value" != "all" ]; then
                  echo "Node: $node, Static Pod: $pod"
                  found=1
              fi
          done <<< "$pods"
      
          if [ "$found" -eq 0 ]; then
              echo "No static pods found with GPU isolation enabled."
          else
              echo "Note: The above nodes will automatically isolate faulty GPUs via the NVIDIA Device Plugin."
          fi
      }
      
      check_gpu_isolation_enabled

      Expected output:

      Checking nodes with NVIDIA Device Plugin deployed as static pods that have GPU isolation enabled (i.e., DP_DISABLE_HEALTHCHECKS is not set to 'all')...
      Node: cn-beijing.192.168.XXX.XXX, Static Pod: nvidia-device-plugin-cn-beijing.192.168.XXX.XXX
      Note: The above nodes will automatically isolate faulty GPUs via NVIDIA Device Plugin.

      If the output contains Note: The above nodes will automatically isolate faulty GPUs via NVIDIA Device Plugin., the GPU isolation feature is enabled.

2. Disable the GPU isolation feature by upgrading the add-on.

   The native GPU isolation feature is disabled by default in the latest version of the NVIDIA Device Plugin add-on. Upgrade the NVIDIA Device Plugin to the latest version to apply this change.