GPU stability best practices
As technologies like AI, deep learning, scientific computing, and big data processing advance, GPUs have become a critical component for high-performance computing. To ensure your systems run smoothly, Alibaba Cloud offers an inspection feature to detect hardware faults early and a self-service diagnostics feature to help you quickly pinpoint problems.
This topic applies only to the Linux operating system.
Use cases
Diagnose GPU-accelerated instance issues
Use the self-service diagnostics feature for Elastic Compute Service (ECS) to diagnose potential issues on your GPU-accelerated instances and quickly locate problems.
GPU O&M
If Alibaba Cloud detects a GPU hardware issue, you will receive an O&M event notification. You can then respond to the ECS system event to automatically resolve the fault.
You can use the self-service diagnostics feature or call the ReportInstancesStatus API operation to report exceptions on one or more GPU-accelerated instances. After receiving the report, Alibaba Cloud checks the instances for exceptions. If an exception is confirmed, an O&M event notification is sent to you. You can then respond to the system event to automatically resolve the fault.
ImportantResponding to a system event restarts the instance, which can interrupt your services. We recommend you restart the service during off-peak hours using one of the following methods:
Respond to the system event directly within the scheduled window.
Diagnose GPU hardware
A running GPU-accelerated instance might have faults or security risks, such as hardware faults or driver errors. You can use the following methods to diagnose the instance:
Run a GPU Health Check from the ECS console to diagnose issues with the GPUs or drivers on your instance.
Use a Cloud Assistant plug-in for a comprehensive diagnosis of the GPU and driver status. This helps you quickly identify common problems, such as GPU or driver exceptions. If an exception is detected, the system can automatically take O&M actions, such as sending you a notification.
Diagnose GPUs by using Troubleshooting
In the upper-left corner of the page, select a region and resource group.
On the Troubleshooting tab, specify the troubleshooting type, check item, instance ID, and troubleshooting period, and then click Check Now.
Set the troubleshooting type to Instance Device Check and the check item to GPU Health Check.
After the diagnosis is complete, view the diagnostic report.
The following table describes the information in the diagnostic report.
Item
Description
Diagnostic result
If all items are normal, the report indicates that no exceptions were found on the instance.
If any items are abnormal, the report indicates the number of exceptions found, displays the details, and provides corresponding solutions. You can follow the recommended solutions to fix the issues.
Diagnostic item details
Includes checks for GPU device and driver status, NVIDIA Xid tool exceptions, GPU memory status, and
Fabricmanagercomponent exceptions. Severity levels are Critical, Warning, and Normal.Basic diagnostic information
Includes the Resource ID, Report ID, and diagnosis time.
Diagnose GPUs by using Cloud Assistant
In the upper-left corner of the page, select a region and resource group.
On the ECS Instances tab, find the instance in the list and click Run Command in the Actions column.
In the Create Command panel, configure the parameters in the Command Information section.
The key parameters are described below. Keep the default values for other parameters. For more information, see Create a command.
Set Timeout to
180seconds.ImportantYou must configure the parameters exactly as described below. Otherwise, the Cloud Assistant command may fail to run.
Command Type: Shell.
Command content: Copy the following command. For more examples of shell commands, see View instance system configurations.
if acs-plugin-manager --list --local | grep ACS-ECS-GpuCheck > /dev/null 2>&1 then acs-plugin-manager --remove --plugin ACS-ECS-GpuCheck fi acs-plugin-manager --exec --plugin ACS-ECS-GpuCheckTimeout: The timeout period for command execution. If a command execution task times out, Cloud Assistant forcibly terminates the task process.
NoteTimeout Period supports only positive integers, in seconds. Valid values: 10 to 86400 seconds (24 hours).
Click Run to diagnose the health of the GPU-accelerated instance with the Cloud Assistant command.
If the execution result shows that all check items are in the
OKstate, it indicates that no GPU anomalies were found.time="2024-07-18T10:25:19+08:00" level=info msg="gpu_check output:" GPU PCI Slot: 00:07.0, Diagnosis result: 00:07.0_1321122071797_0_0 * Double Bit Error Check – OK * Info Rom Corrupted Check – OK * eRDMA Incorrect Check – OK * Kernel Upgrade Check – OK * Fabricmanager running Check – OK * Power Cable Error Check – OK * GPU Device Lost Check – OK * GPU Driver Install Check – OK * GPU Xid Error Check – OK GPU PCI Slot: 00:08.0, Diagnosis result: 00:08.0_1321122085127_0_0 * Power Cable Error Check – OK * GPU Device Lost Check – OK * GPU Driver Install Check – OKIf one or more check items are in the
Failedstate, it indicates that GPU anomalies were found.time="2024-07-18T10:26:24+08:00" level=info msg="Begin to gpu health check" time="2024-07-18T10:26:25+08:00" level=info msg="gpu_check output:" GPU PCI Slot: 00:07.0, Diagnosis result: 00:07.0_N/A_10100000_0 * Power Cable Error Check – OK * Double Bit Error Check – OK * GPU Driver Install Check – Failed * Kernel Upgrade Check – Failed * GPU Device Lost Check – OK * Info Rom Corrupted Check – OK * eRDMA Incorrect Check – OK * GPU Xid Error Check – OK * Fabricmanager running Check – OK time="2024-07-18T10:26:25+08:00" level=info msg="Detailed logs are located in /var/log/check_gpu.tar.gz" time="2024-07-18T10:26:25+08:00" level=info msg="Complete GPU health check"
The following table describes the diagnostic items in the output.
Item
Description
Actions
Double Bit Error Check
Checks the GPU for double-bit errors.
Prompts you to restart the instance based on the number of errors.
Info Rom Corrupted Check
Checks the GPU's infoROM firmware.
Sends you an O&M event notification.
eRDMA Incorrect Check
Checks the status of the GPU's elastic RDMA interface (ERI).
Sends you an O&M event notification.
Kernel Upgrade Check
Checks for driver exceptions caused by a kernel upgrade.
Uninstall and then reinstall the driver.
Fabricmanager Running Check
Checks the running status of the
Fabricmanagercomponent.Install or start the
Fabricmanagercomponent service.Power Cable Error Check
Checks the GPU power cable and power supply status.
Sends you an O&M event notification.
GPU Device Lost Check
Checks if the GPU device is lost.
Sends you an O&M event notification.
GPU Driver Install Check
Checks the installation status of the GPU driver.
Install the driver.
GPU Xid Error Check
Checks for GPU Xid errors.
Prompts you to restart the instance based on the specific XID error.