One-click GPU health check using the Cloud Assistant plug-in-Elastic GPU Service(EGS)-阿里云帮助中心

You can use the Cloud Assistant plug-in to run a comprehensive diagnosis of the GPU and driver status on your instance. This helps you quickly detect common issues, such as hardware or driver anomalies. If an anomaly is detected, the system automatically performs an O&M action, such as sending a notification.

Procedure

Note

This topic applies to diagnosing GPU instances running a Linux operating system. The Cloud Assistant plug-in is pre-installed on these instances by default. For more information about Cloud Assistant, see Cloud Assistant overview.

Log in to the ECS console.
In the left-side navigation pane, choose Maintenance & Monitoring > Cloud Assistant.
In the upper-left corner of the page, select the region of your target GPU instance.
On the ECS Instance tab, find the target instance in the list and click Run Command in the Actions column.
In the Create Command panel, configure the parameters in the Command Information section.

Configure the key parameters as described below and keep the default values for other parameters. For more information, see Create a command.

Important
Set the parameters to the exact values shown below. Otherwise, the command may fail.

Command Type: Select Shell.

Command content: Paste the following command. For other Shell command examples, see View the system configurations of an instance.
```
if acs-plugin-manager --list --local | grep ACS-ECS-GpuCheck > /dev/null 2>&1
then
    acs-plugin-manager --remove --plugin ACS-ECS-GpuCheck
fi
acs-plugin-manager --exec --plugin ACS-ECS-GpuCheck
```
Timeout Period: Set the value to 180. If the command does not complete within this period, Cloud Assistant forcibly terminates the task process.

Note
The Timeout Period, in seconds, must be an integer between 10 and 86400 (24 hours).

Click Run to execute the command.

A status of OK for all check items indicates that no GPU anomalies were detected.

time="2024-07-18T10:25:19+08:00" level=info msg="gpu_check output:"
    GPU PCI Slot: 00:07.0, Diagnosis result: 00:07.0_1321122071797_0_0
    * Double Bit Error Check - OK
    * Info Rom Corrupted Check - OK
    * eRDMA Incorrect Check - OK
    * Kernel Upgrade Check - OK
    * Fabricmanager running Check - OK
    * Power Cable Error Check - OK
    * GPU Device Lost Check - OK
    * GPU Driver Install Check - OK
    * GPU Xid Error Check - OK
    GPU PCI Slot: 00:08.0, Diagnosis result: 00:08.0_1321122085127_0_0
    * Power Cable Error Check - OK
    * GPU Device Lost Check - OK
    * GPU Driver Install Check - OK

A status of Failed for one or more check items indicates a GPU anomaly. For example, a Failed status for GPU Driver Install Check indicates a driver issue.

time="2024-07-18T10:26:24+08:00" level=info msg="Begin to gpu health check"
time="2024-07-18T10:26:25+08:00" level=info msg="gpu_check output:"
GPU PCI Slot: 00:07.0, Diagnosis result: 00:07.0_N/A_10100000_0
 * Power Cable Error Check - OK
 * Double Bit Error Check - OK
 * GPU Driver Install Check - Failed
 * Kernel Upgrade Check - Failed
 * GPU Device Lost Check - OK
 * Info Rom Corrupted Check - OK
 * eRDMA Incorrect Check - OK
 * GPU Xid Error Check - OK
 * Fabricmanager running Check - OK
time="2024-07-18T10:26:25+08:00" level=info msg="Detailed logs are located in /var/log/check_gpu.tar.gz"
time="2024-07-18T10:26:25+08:00" level=info msg="Complete GPU health check"

Diagnostic items and troubleshooting

The following table describes the diagnostic items for the GPU health check.

Item	Description	Actions
Double Bit Error Check	Checks for double-bit errors on the GPU.	The system prompts you to restart the instance based on the number of errors.
Info Rom Corrupted Check	Checks the GPU's infoROM firmware information.	The system sends an O&M notification.
eRDMA Incorrect Check	Checks the status of the eRDMA network interface on the GPU.	The system sends an O&M notification.
Kernel Upgrade Check	Checks for driver anomalies caused by a kernel upgrade.	Uninstall and then reinstall the current driver.
Fabricmanager running Check	Checks the running status of the Fabricmanager component.	Install or start the Fabricmanager component service.
Power Cable Error Check	Checks the GPU power cable and power supply status.	The system sends an O&M notification.
GPU Device Lost Check	Checks whether the GPU device is detected by the operating system.	The system sends an O&M notification.
GPU Driver Install Check	Checks the installation status of the GPU driver.	Install the driver.
GPU Xid Error Check	Checks for GPU Xid error anomalies.	The system prompts you to restart the instance based on the specific XID error.

上一篇: Diagnose a GPU-accelerated instance with Self-service Troubleshooting 下一篇: Run DCGM diagnostics using Cloud Assistant plugin