How to upgrade the cGPU module on a node using commands and the console-Container Service for Kubernetes(ACK)-阿里云帮助中心

Nodes in an ACK cluster must have the cGPU module installed to enable GPU sharing and scheduling. This topic shows you how to upgrade the cGPU module on a node by using the ACK console and commands.

Step 1: Upgrade the component

Cluster type

Component upgrade method

ACK managed cluster Pro
ACK Edge cluster Pro

To upgrade the ack-ai-installer component, see Upgrade the GPU sharing and scheduling component.

ACK dedicated cluster

To upgrade the ack-cgpu component:

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Helm.
On the Helm page, find the ack-cgpu component, click Update in the Actions column, select a Version, and then click OK.

Step 2: Upgrade existing nodes

Important

Stop all GPU workloads on the node during the upgrade.
Upgrade one node first. After you verify that the GPU workloads run as expected, upgrade the other GPU nodes in batches.
This method resets the system disk of the node. If the system disk contains data, create a backup first.

1. Remove and re-add the node

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Nodes.
On the Nodes page, select the cGPU node to upgrade and click Batch Remove. In the Remove Node dialog box, select Drain Node.
Re-add the removed node to its original node pool. For more information, see Add existing nodes.

Important
Add the node automatically. The manual addition method does not reset the node.

2. Verify the result

Run the following command to check the status of the cgpu-installer Pod for the newly added node.

kubectl get po -l name=cgpu-installer -n kube-system -o wide

Expected output:

NAME                   READY   STATUS    RESTARTS   AGE    IP                NODE                         NOMINATED NODE   READINESS GATES
cgpu-installer-*****   1/1     Running   0          4d2h   192.168.XXX.XX1   cn-beijing.192.168.XXX.XX1   <none>           <none>
cgpu-installer-**2     1/1     Running   0          4d2h   192.168.XXX.XX2   cn-beijing.192.168.XXX.XX2   <none>           <none>
cgpu-installer-**3     1/1     Running   0          4d2h   192.168.XXX.XX3   cn-beijing.192.168.XXX.XX3   <none>           <none>

Run the following command to open a shell in the cgpu-installer-****** Pod.
```
kubectl exec -ti cgpu-installer-xxxxx -n kube-system -- bash
```
Run the following command to check the current cGPU version.
```
nsenter -t 1 -i -p -n -u -m -- cat /proc/cgpu_km/version
```
Example output:
```
1.5.16
```
Note
For the latest cGPU version, see ack-ai-installer.

cGPU version compatibility

NVIDIA driver compatibility

cGPU version

Compatible NVIDIA drivers

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

Supported:

460 series
470 series
510 series
515 series
525 series
535 series
550 series
560 series
565 series
570 series
575 series

1.5.6

1.5.5

1.5.3

1.5.2

1.0.10

1.0.9

1.0.8

1.0.7

1.0.6

1.0.5

Supported:

460 series
470 series <= 470.161.03
510 series <= 510.108.03
515 series <= 515.86.01
525 series <= 525.89.03

Not supported:

535 series
550 series
560 series
565 series
570 series
575 series

1.0.3

0.8.17

0.8.13

Supported:

460 series
470 series <= 470.161.03

Not supported:

510 series
515 series
525 series
535 series
550 series
560 series
565 series
570 series
575 series

Instance family compatibility

cGPU version

Compatible instance families

1.5.20

1.5.19

Supported:

gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e
gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e
gn8t / ebmgn8t
gn8is / gn8v / ebmgn8is / ebmgn8v
gn8ia / ebmgn8ia
ebmgn9t

1.5.18

1.5.17

1.5.16

1.5.15

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

Supported:

gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e
gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e
gn8t / ebmgn8t
gn8is / gn8v / ebmgn8is / ebmgn8v
gn8ia / ebmgn8ia

Not supported:

ebmgn9t

1.5.8

1.5.7

Supported:

gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e
gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e
gn8t / ebmgn8t
gn8is / gn8v / ebmgn8is / ebmgn8v

Not supported:

gn8ia / ebmgn8ia
ebmgn9t

1.5.6

1.5.5

1.5.3

1.5.2

1.0.10

1.0.9

1.0.8

1.0.7

1.0.6

1.0.5

1.0.3

Supported:

gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e
gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e

Not supported:

gn8t / ebmgn8t
gn8is / gn8v / ebmgn8is / ebmgn8v
gn8ia / ebmgn8ia
ebmgn9t

0.8.17

0.8.13

Supported:

gn6i / gn6e / gn6v / gn6t / ebmgn6i / ebmgn6t / ebmgn6e

Not supported:

gn7i / gn7 / gn7e / ebmgn7i / ebmgn7e
gn8t / ebmgn8t
gn8is / gn8v / ebmgn8is / ebmgn8v
gn8ia / ebmgn8ia
ebmgn9t

nvidia-container-toolkit compatibility

cGPU version

Compatible nvidia-container-toolkit

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.6

1.5.5

1.5.3

1.5.2

1.0.10

Supported:

nvidia-container-toolkit <= 1.10
nvidia-container-toolkit: 1.11 to 1.17

1.0.9

1.0.8

1.0.7

1.0.6

1.0.5

1.0.3

0.8.17

0.8.13

Supported:

nvidia-container-toolkit <= 1.10

Not supported:

nvidia-container-toolkit: 1.11 to 1.17

Kernel version compatibility

cGPU version

Compatible kernel versions

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

Supported:

kernel 3.x
kernel 4.x
kernel 5.x <= 5.15

1.5.8

Supported:

kernel 3.x
kernel 4.x
kernel 5.x <= 5.10

1.5.7

1.5.6

1.5.5

1.5.3

1.5.2

1.0.10

1.0.9

1.0.8

1.0.7

1.0.6

1.0.5

1.0.3

Supported:

kernel 3.x
kernel 4.x
kernel 5.x <= 5.1

0.8.17

Supported:

kernel 3.x
kernel 4.x
kernel 5.x <= 5.0

0.8.13

0.8.12

0.8.10

Supported:

kernel 3.x
kernel 4.x

Not supported:

kernel 5.x