NVIDIA GPU (Tesla) driver load failure after kernel update-Alibaba Cloud Linux(Alinux)-阿里云帮助中心

Symptoms

After you upgrade the operating system kernel on a GPU-accelerated instance, the NVIDIA GPU (Tesla) driver fails to load. This happens because the NVIDIA kernel object (KO) built for the old kernel cannot be inserted into the new kernel, preventing the driver from functioning. The error message is similar to the following:

$sudo insmod ./nvidia.ko
insmod: ERROR: could not insert module ./nvidia.ko: Invalid module format

Causes

Possible causes of this issue include:

A kABI mismatch between the old and new kernels prevents the pre-existing NVIDIA GPU (Tesla) driver from loading.
The new kernel package cannot create a symbolic link to the driver's kernel object (KO) during installation because the KO is not in the default /lib/modules/$(uname -r)/extra directory.

Solutions

Choose a solution based on whether the kernel's kAPI is affected by the update:

If the kAPI is not affected by the kernel update, use DKMS to automatically build the NVIDIA GPU (Tesla) driver.
If the kAPI is affected by the kernel update and DKMS cannot automatically build the driver, reinstall the NVIDIA GPU (Tesla) driver.

Use DKMS to auto-build the NVIDIA GPU (Tesla) driver

Install DKMS and register the NVIDIA GPU (Tesla) driver.
1. Connect to the GPU-accelerated instance.
  
  This topic uses a gn7i instance that runs Alibaba Cloud Linux 3 as an example. For more information, see Connect to a Linux instance by using Workbench.
2. Install DKMS on the GPU-accelerated instance.
```
sudo yum install dkms
```
3. Manually install the NVIDIA GPU (Tesla) driver.
  
  For more information, see Manually install a Tesla driver on a GPU-accelerated compute-optimized Linux instance.
  
  During the installation, note the following:
  - When prompted whether to register the kernel module sources with DKMS, select Yes.
```
Would you like to register the kernel module sources with DKMS? This will allow DKMS to automatically build a new module, if you install a different kernel later.
<Yes>  <No>
```
  - After you select Yes, a warning about a registration failure may appear, as shown in the following output. You can ignore this warning and click OK.
```
WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system.  If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver.
```
  - Choose whether to install the NVIDIA 32-bit compatibility libraries.
4. Run the following command to check the status of DKMS.
```
sudo dkms status
```
  The following output indicates that DKMS is installed successfully.
```
[ecs-userxxx 2Z ~]$ sudo dkms status
nvidia/470.141.03, 5.10.134-16.3.al8.x86_64, x86_64: installed
```
5. Run the ls command to verify that the NVIDIA GPU (Tesla) driver source files are stored in the /usr/src/nvidia-${nvidia_driver_version} directory.
  
  In this example, nvidia-${nvidia_driver_version} is nvidia-470.141.03. Replace it with your actual driver version.
```
[ecs-user@xxx ~]$ ls  /usr/src/nvidia-470.141.03
common  conftest.sh  dkms.conf  Kbuild  Makefile  nvidia  nvidia-drm  nvidia-modeset  nvidia-peermem  nvidia-uvm
```
  Note
  By default, the NVIDIA GPU (Tesla) driver stores its source code and related files in the /usr/src/nvidia-${nvidia_driver_version} directory. This allows DKMS to automatically recompile and install the driver's kernel module after a kernel update.

Install the new kernel to trigger DKMS to automatically build the NVIDIA GPU (Tesla) driver.

This example uses the kernel version 5.10.134-15.al8. Replace it with the kernel version that you require.

Important

You must install the kernel-devel package before the kernel or kernel-core package. The kernel package installation triggers DKMS, but the build process requires files from the kernel-devel package. If you install these packages out of order, the automatic build fails, and you must trigger the build manually. For more information, see Step 3. (Required if applicable) Manually trigger DKMS to build the NVIDIA GPU (Tesla) driver.

Run the following command to install the kernel-devel package for the new kernel.

sudo rpm -ivh kernel-devel-5.10.134-15.al8.x86_64.rpm --force

[ecs-user@xxx ~]$ sudo rpm -ivh kernel-devel-5.10.134-15.al8.x86_64.rpm --force
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:kernel-devel-5.10.134-15.al8     ################################# [100%]

Install the kernel or kernel-core package.

This example demonstrates installing the kernel package. For Alibaba Cloud Linux 3, you must install the kernel-core package instead by running sudo rpm -ivh kernel-core-5.10.134-15.al8.x86_64.rpm --force.

sudo rpm -ivh kernel-5.10.134-15.al8.x86_64.rpm --force

sudo rpm -ivh kernel-5.10.134-15.al8.x86_64.rpm --force
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:kernel-5.10.134-15.al8           ################################# [100%]

Run the following command to check whether the NVIDIA GPU (Tesla) driver was successfully built for the new kernel.

find /lib/modules/5.10.134-15.al8.x86_64/ -name *nvidia*

[ecs-user@iZxxxxxrjl6Z ~]$ find /lib/modules/5.10.134-15.al8.x86_64/ -name *nvidia*
/lib/modules/5.10.134-15.al8.x86_64/extra/nvidia-drm.ko
/lib/modules/5.10.134-15.al8.x86_64/extra/nvidia-uvm.ko
/lib/modules/5.10.134-15.al8.x86_64/extra/nvidia.ko
/lib/modules/5.10.134-15.al8.x86_64/extra/nvidia-peermem.ko
/lib/modules/5.10.134-15.al8.x86_64/extra/nvidia-modeset.ko

Run sudo dkms status to check if DKMS has a record for the new kernel.

[ecs-user@ixxxxxxxxxxrjl6Z ~]$ sudo dkms status
nvidia/470.141.03, 5.10.134-15.al8.x86_64, x86_64: installed    # Target kernel version
nvidia/470.141.03, 5.10.134-16.3.al8.x86_64, x86_64: installed
nvidia/470.141.03, 5.10.134-17.1.al8.x86_64, x86_64: installed

(Required if applicable) You must manually trigger DKMS to build the NVIDIA GPU (Tesla) driver if you installed the kernel or kernel-core package before the kernel-devel package.

Run the following command to build the NVIDIA GPU (Tesla) driver.

sudo dkms build -m nvidia -v ${nvidia_driver_version} -k ${new_kernel_version} --force

Parameters:

${nvidia_driver_version}: Replace this with the version of your NVIDIA GPU (Tesla) driver, such as 470.141.03.
${new_kernel_version}: Replace this with the version of the new kernel, such as 5.10.134-15.al8.x86_64.

[ecs-t...xxx...zd34Z ~]$ sudo dkms build -m nvidia -v 470.141.03 -k 5.10.134-15.al8.x86_64 --force
Sign command: /lib/modules/5.10.134-15.al8.x86_64/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub
Building module:
Cleaning build area...
Building module(s).....
Signing module /var/lib/dkms/nvidia/470.141.03/build/nvidia.ko
Signing module /var/lib/dkms/nvidia/470.141.03/build/nvidia-uvm.ko
Signing module /var/lib/dkms/nvidia/470.141.03/build/nvidia-modeset.ko
Signing module /var/lib/dkms/nvidia/470.141.03/build/nvidia-drm.ko
Signing module /var/lib/dkms/nvidia/470.141.03/build/nvidia-peermem.ko
Cleaning build area...
DKMS: build completed.

Run the following command to install the newly built NVIDIA GPU (Tesla) driver.

sudo dkms install -m nvidia -v ${nvidia_driver_version} -k ${new_kernel_version} --force

[ecs-us xxx d34Z ~]$ sudo dkms install -m nvidia -v 470.141.03 -k 5.10.134-15.al8.x86_64 --force
nvidia.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.10.134-15.al8.x86_64/extra/
nvidia-uvm.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.10.134-15.al8.x86_64/extra/
nvidia-modeset.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.10.134-15.al8.x86_64/extra/
nvidia-drm.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.10.134-15.al8.x86_64/extra/
nvidia-peermem.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.10.134-15.al8.x86_64/extra/
Adding any weak-modules
depmod...
DKMS: install completed.
   - Installing to /lib/modules/5.10.134-15.al8.x86_64/extra/
nvidia-peermem.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.10.134-15.al8.x86_64/extra/
Adding any weak-modules
depmod...

Run the following command to check whether the NVIDIA GPU (Tesla) driver is installed in the new kernel's directory.

find /lib/modules/5.10.134-15.al8.x86_64/ -name *nvidia*
/lib/modules/5.10.134-15.al8.x86_64/extra/nvidia.ko
/lib/modules/5.10.134-15.al8.x86_64/extra/nvidia-modeset.ko
/lib/modules/5.10.134-15.al8.x86_64/extra/nvidia-drm.ko
/lib/modules/5.10.134-15.al8.x86_64/extra/nvidia-peermem.ko

Run sudo dkms status to check if DKMS has a record for the new kernel.

[ecs-u xxx pzd34Z ~]$ sudo dkms status
nvidia/470.141.03, 5.10.134-15.al8.x86_64, x86_64: installed
nvidia/470.141.03, 5.10.134-16.3.al8.x86_64, x86_64: installed

Reinstall the NVIDIA GPU (Tesla) driver

If the kernel update changes the kAPI, DKMS cannot automatically build and install the NVIDIA GPU (Tesla) driver. In this case, you must reinstall the driver. For more information, see Manually install a Tesla driver on a GPU-accelerated compute-optimized Linux instance.