Run DCGM diagnostics using the Cloud Assistant plugin

Overview

NVIDIA Data Center GPU Manager (DCGM) is a toolkit for managing and monitoring NVIDIA GPUs in cluster environments. It provides health monitoring, diagnostics, and system alerts.

Benefits:

• No pre-installation required: The plugin automatically downloads the required dcgm libraries and diagnostic components, even if DCGM is not installed.

• Fully automated: The plugin detects the environment, downloads dependencies, runs diagnostics, and outputs results in a single click.

Prerequisites

Make sure that the following prerequisites are met:

• Instance type: An Elastic GPU Service instance.

• Cloud Assistant: The Cloud Assistant plugin is installed on the instance.

Procedure

1. Log on to the Linux instance using Workbench.

2. Run the diagnostic command:

   Run acs-plugin-manager with the diagnostic plugin. Replace {Level} with the desired diagnostic level.

   acs-plugin-manager --exec --plugin ACS-ECS-DcgmDiagnostics -p "--diag-level {Level}"

   Diagnostic levels ({Level}):

   Select one of the following levels:

   Level	Description	Use case
   short	Basic, fast health check.	Quick daily checks. Shortest runtime.
   medium	Medium-length diagnostics with additional stress tests.	Suspected performance issues.
   long	Thorough, deep diagnostics with long-duration stress tests.	Critical failures or hardware acceptance before delivery.

   Higher levels take longer and consume significant GPU resources. A long-level diagnostic may take several minutes. DCGM Diagnostics.

Sample output

The following example shows the output of a long-level diagnostic on an ecs.gn7i-c8g1.2xlarge instance:

+-------------------------+------------------------------------+
| Diagnostic              | Result                             |
+=========================+====================================+
| Metadata                |                                    |
+-------------------------+------------------------------------+
| DCGM Version            | 4.5.2                              |
+-------------------------+------------------------------------+
| Driver Version Detected | 580.126.09                         |
+-------------------------+------------------------------------+
| software                | Pass                               |
|                         |                                    |
|                         | GPU0 0000:00:03.0 NVIDIA A10: Pass |
+-------------------------+------------------------------------+
| memory                  | Pass                               |
|                         |                                    |
|                         | GPU0 0000:00:03.0 NVIDIA A10: Pass |
+-------------------------+------------------------------------+
| diagnostic              | Pass                               |
|                         |                                    |
|                         | GPU0 0000:00:03.0 NVIDIA A10: Pass |
+-------------------------+------------------------------------+
| nvbandwidth             | Skip                               |
|                         |                                    |
|                         | GPU0 0000:00:03.0 NVIDIA A10: Skip |
+-------------------------+------------------------------------+
| pcie                    | Pass                               |
|                         |                                    |
|                         | GPU0 0000:00:03.0 NVIDIA A10: Pass |
+-------------------------+------------------------------------+
| memory_bandwidth        | Pass                               |
|                         |                                    |
|                         | GPU0 0000:00:03.0 NVIDIA A10: Pass |
+-------------------------+------------------------------------+
| targeted_stress         | Pass                               |
|                         |                                    |
|                         | GPU0 0000:00:03.0 NVIDIA A10: Pass |
+-------------------------+------------------------------------+
| targeted_power          | Pass                               |
|                         |                                    |
|                         | GPU0 0000:00:03.0 NVIDIA A10: Pass |
+-------------------------+------------------------------------+

FAQ

• How do I run diagnostics on multiple instances automatically?

  Use the Cloud Assistant public command ACS-ECS-DCGMDiagnostic-for-linux.sh to run DCGM diagnostics in batch on multiple instances.

• Do all GPU instance types support DCGM diagnostics?

  Tesla GPU instances support all DCGM diagnostic levels. Other GPU instance types support only Level 1 (short) diagnostics. NVIDIA DCGM Feature Overview.