Diagnose a GPU-accelerated instance with Self-service Troubleshooting
GPU-accelerated instances may encounter potential failures or security risks during runtime, such as GPU card malfunctions or driver errors. The ECS console provides a GPU Health Check feature that lets you diagnose issues with your instance's GPU card or driver. This helps you identify and resolve potential issues quickly.
Procedure
Before you begin, make sure the GPU-accelerated instance is in the Running state.
-
Go to the Self-service Troubleshooting page. In the upper-left corner of the page, select the region of the target GPU-accelerated instance.
-
In the and click Initiate Diagnosis.
-
On the Create Diagnostic Task page, select the GPU-accelerated instance that you want to diagnose and specify a time range. Click Initiate Diagnosis.
-
After the diagnosis is complete, view the diagnostic report and follow the recommended solutions to resolve any issues.
The diagnostic report shows the results for GPU Health Diagnosis. In this example, out of 6 total diagnostic items, the GPU driver status check returns a warning. This indicates that the GPU driver is not installed, which prevents the instance from using the GPU card. Refer to Install Tesla drivers to fix the issue. The other 5 items (GPU device and driver status check, NVIDIA Xid tool exception check, GPU NVLink status check, GPU memory status check, and Fabricmanager component exception check) all passed.