Upgrade the Docker runtime of a GPU node-Container Service for Kubernetes(ACK)-阿里云帮助中心

Upgrade Docker to 19.03.5 with nvidia-container-runtime to enable shared GPU resource isolation on cluster nodes.

Background information

nvidia-container-runtime allows you to build and run GPU-accelerated Docker containers. It automatically configures containers to use NVIDIA GPUs.

Prerequisites

Ensure the following:

A GPU node running CentOS or Alibaba Cloud Linux 2
kubectl is connected to the ACK cluster. See Connect to a cluster with kubectl.
The GPU node name from kubectl get nodes. Replace <NODE_NAME> with this value in the following steps

Procedure

Cordon the node

Run the following command on the master node to mark the target node as unschedulable and block new pod scheduling during the upgrade.

kubectl cordon <NODE_NAME>

Drain the node

Migrate existing pods to other available nodes.

kubectl drain <NODE_NAME> --ignore-daemonsets --delete-local-data --force

Log in to the GPU node and stop services

Stop kubelet and Docker before upgrading.

service kubelet stop
docker rm -f $(docker ps -aq)
service docker stop

Remove Docker and nvidia-container-runtime

Remove the existing Docker runtime and nvidia-container-runtime packages.

yum remove -y docker-ce docker-ce-cli containerd
yum remove -y nvidia-container-runtime*  libnvidia-container*

Back up daemon.json

View the current configuration, then move the file to /tmp as a backup.

View the current configuration. Example output:

   cat /etc/docker/daemon.json

   {
       "default-runtime": "nvidia",
       "runtimes": {
           "nvidia": {
               "path": "/usr/bin/nvidia-container-runtime",
               "runtimeArgs": []
           }
       },
       "exec-opts": ["native.cgroupdriver=systemd"],
       "log-driver": "json-file",
       "log-opts": {
           "max-size": "100m",
           "max-file": "10"
       },
       "bip": "169.254.123.1/24",
       "oom-score-adjust": -1000,
       "storage-driver": "overlay2",
       "storage-opts":["overlay2.override_kernel_check=true"],
       "live-restore": true
   }

Move the file to /tmp:
```
   mv /etc/docker/daemon.json /tmp
```

Install Docker 19.03.5

Download and install Docker 19.03.5.

VERSION=19.03.5
URL=http://aliacs-k8s-cn-beijing.oss-cn-beijing.aliyuncs.com/public/pkg/docker/docker-${VERSION}.tar.gz
curl -ssL $URL -o /tmp/docker-${VERSION}.tar.gz
cd /tmp
tar -xf docker-${VERSION}.tar.gz
cd /tmp/pkg/docker/${VERSION}/rpm
yum localinstall -y $(ls .)

Install nvidia-container-runtime 3.13.0

Download and install nvidia-container-runtime 3.13.0.

sudo cd /tmp
sudo yum install -y unzip
sudo wget https://aliacs-k8s-cn-hongkong.oss-cn-hongkong.aliyuncs.com/public/pkg/nvidia-container-runtime/nvidia-container-runtime-3.13.0-linux-amd64.tar.gz
sudo tar -xvf nvidia-container-runtime-3.13.0-linux-amd64.tar.gz
sudo yum -y -q --nogpgcheck localinstall pkg/nvidia-container-runtime/3.13.0/common/*

Restore daemon.json

Restore the backed-up configuration file.

Restore the file:

   mv /tmp/daemon.json  /etc/docker/daemon.json

Verify the configuration. Example output:

   cat /etc/docker/daemon.json

   {
       "default-runtime": "nvidia",
       "runtimes": {
           "nvidia": {
               "path": "/usr/bin/nvidia-container-runtime",
               "runtimeArgs": []
           }
       },
       "exec-opts": ["native.cgroupdriver=systemd"],
       "log-driver": "json-file",
       "log-opts": {
           "max-size": "100m",
           "max-file": "10"
       },
       "bip": "169.254.123.1/24",
       "oom-score-adjust": -1000,
       "storage-driver": "overlay2",
       "storage-opts":["overlay2.override_kernel_check=true"],
       "live-restore": true
   }

Start Docker and kubelet

Start Docker and kubelet.

service docker start
service kubelet start

Uncordon the node

Mark the node as schedulable to resume pod scheduling.

kubectl uncordon <NODE_NAME>

Restart the cGPU installer

Restart the cGPU installer on the node.

docker ps |grep cgpu-installer | awk '{print $1}' | xargs docker rm -f