GPU stability best practices

Updated at:

As technologies like AI, deep learning, scientific computing, and big data processing advance, GPUs have become a critical component for high-performance computing. To ensure your systems run smoothly, Alibaba Cloud offers an inspection feature to detect hardware faults early and a self-service diagnostics feature to help you quickly pinpoint problems.

Important

This topic applies only to the Linux operating system.

Use cases

  • Diagnose GPU-accelerated instance issues

    Use the self-service diagnostics feature for Elastic Compute Service (ECS) to diagnose potential issues on your GPU-accelerated instances and quickly locate problems.

  • GPU O&M

    • If Alibaba Cloud detects a GPU hardware issue, you will receive an O&M event notification. You can then respond to the ECS system event to automatically resolve the fault.

    • You can use the self-service diagnostics feature or call the ReportInstancesStatus API operation to report exceptions on one or more GPU-accelerated instances. After receiving the report, Alibaba Cloud checks the instances for exceptions. If an exception is confirmed, an O&M event notification is sent to you. You can then respond to the system event to automatically resolve the fault.

    Important

    Responding to a system event restarts the instance, which can interrupt your services. We recommend you restart the service during off-peak hours using one of the following methods:

Diagnose GPU hardware

A running GPU-accelerated instance might have faults or security risks, such as hardware faults or driver errors. You can use the following methods to diagnose the instance:

  • Run a GPU Health Check from the ECS console to diagnose issues with the GPUs or drivers on your instance.

  • Use a Cloud Assistant plug-in for a comprehensive diagnosis of the GPU and driver status. This helps you quickly identify common problems, such as GPU or driver exceptions. If an exception is detected, the system can automatically take O&M actions, such as sending you a notification.

Diagnose GPUs by using Troubleshooting

  1. Go to ECS Console - Troubleshooting.

  2. In the upper-left corner of the page, select a region and resource group.

  3. On the Troubleshooting tab, specify the troubleshooting type, check item, instance ID, and troubleshooting period, and then click Check Now.

    Set the troubleshooting type to Instance Device Check and the check item to GPU Health Check.

    After the diagnosis is complete, view the diagnostic report.

    The following table describes the information in the diagnostic report.

    Item

    Description

    Diagnostic result

    • If all items are normal, the report indicates that no exceptions were found on the instance.

    • If any items are abnormal, the report indicates the number of exceptions found, displays the details, and provides corresponding solutions. You can follow the recommended solutions to fix the issues.

    Diagnostic item details

    Includes checks for GPU device and driver status, NVIDIA Xid tool exceptions, GPU memory status, and Fabricmanager component exceptions. Severity levels are Critical, Warning, and Normal.

    Basic diagnostic information

    Includes the Resource ID, Report ID, and diagnosis time.

Diagnose GPUs by using Cloud Assistant

  1. Go to ECS console - Cloud Assistant.

  2. In the upper-left corner of the page, select a region and resource group.

  3. On the ECS Instances tab, find the instance in the list and click Run Command in the Actions column.

  4. In the Create Command panel, configure the parameters in the Command Information section.

    The key parameters are described below. Keep the default values for other parameters. For more information, see Create a command.

    Set Timeout to 180 seconds.

    Important

    You must configure the parameters exactly as described below. Otherwise, the Cloud Assistant command may fail to run.

    • Command Type: Shell.

    • Command content: Copy the following command. For more examples of shell commands, see View instance system configurations.

      if acs-plugin-manager --list --local | grep ACS-ECS-GpuCheck > /dev/null 2>&1
      then
          acs-plugin-manager --remove --plugin ACS-ECS-GpuCheck
      fi
      acs-plugin-manager --exec --plugin ACS-ECS-GpuCheck
    • Timeout: The timeout period for command execution. If a command execution task times out, Cloud Assistant forcibly terminates the task process.

      Note

      Timeout Period supports only positive integers, in seconds. Valid values: 10 to 86400 seconds (24 hours).

  5. Click Run to diagnose the health of the GPU-accelerated instance with the Cloud Assistant command.

    • If the execution result shows that all check items are in the OK state, it indicates that no GPU anomalies were found.

      time="2024-07-18T10:25:19+08:00" level=info msg="gpu_check output:"
          GPU PCI Slot: 00:07.0, Diagnosis result: 00:07.0_1321122071797_0_0
          * Double Bit Error Check – OK
          * Info Rom Corrupted Check – OK
          * eRDMA Incorrect Check – OK
          * Kernel Upgrade Check – OK
          * Fabricmanager running Check – OK
          * Power Cable Error Check – OK
          * GPU Device Lost Check – OK
          * GPU Driver Install Check – OK
          * GPU Xid Error Check – OK
          GPU PCI Slot: 00:08.0, Diagnosis result: 00:08.0_1321122085127_0_0
          * Power Cable Error Check – OK
          * GPU Device Lost Check – OK
          * GPU Driver Install Check – OK
    • If one or more check items are in the Failed state, it indicates that GPU anomalies were found.

      time="2024-07-18T10:26:24+08:00" level=info msg="Begin to gpu health check"
      time="2024-07-18T10:26:25+08:00" level=info msg="gpu_check output:"
      GPU PCI Slot: 00:07.0, Diagnosis result: 00:07.0_N/A_10100000_0
       * Power Cable Error Check – OK
       * Double Bit Error Check – OK
       * GPU Driver Install Check – Failed
       * Kernel Upgrade Check – Failed
       * GPU Device Lost Check – OK
       * Info Rom Corrupted Check – OK
       * eRDMA Incorrect Check – OK
       * GPU Xid Error Check – OK
       * Fabricmanager running Check – OK
      time="2024-07-18T10:26:25+08:00" level=info msg="Detailed logs are located in /var/log/check_gpu.tar.gz"
      time="2024-07-18T10:26:25+08:00" level=info msg="Complete GPU health check"

    The following table describes the diagnostic items in the output.

    Item

    Description

    Actions

    Double Bit Error Check

    Checks the GPU for double-bit errors.

    Prompts you to restart the instance based on the number of errors.

    Info Rom Corrupted Check

    Checks the GPU's infoROM firmware.

    Sends you an O&M event notification.

    eRDMA Incorrect Check

    Checks the status of the GPU's elastic RDMA interface (ERI).

    Sends you an O&M event notification.

    Kernel Upgrade Check

    Checks for driver exceptions caused by a kernel upgrade.

    Uninstall and then reinstall the driver.

    Fabricmanager Running Check

    Checks the running status of the Fabricmanager component.

    Install or start the Fabricmanager component service.

    Power Cable Error Check

    Checks the GPU power cable and power supply status.

    Sends you an O&M event notification.

    GPU Device Lost Check

    Checks if the GPU device is lost.

    Sends you an O&M event notification.

    GPU Driver Install Check

    Checks the installation status of the GPU driver.

    Install the driver.

    GPU Xid Error Check

    Checks for GPU Xid errors.

    Prompts you to restart the instance based on the specific XID error.