GPU diagnostics

Limitations

• Region limitations

  This feature is supported only in the Chinese mainland and the China (Hong Kong) region.

• Operating system limitations

  Architecture	Operating system
  x86	Alibaba Cloud Linux 3 Alibaba Cloud Linux 2 Ubuntu 22.04 Ubuntu 24.04

• Instance limitations

  This feature is available only for GPU instances.

• Slow node diagnosis limitations

  Supported NCCL versions for slow node diagnosis

  v2.22.3.1
  v2.24.3.1
  v2.26.2.1
  v2.26.5.1
  v2.27.3.1
  v2.27.6.1
  v2.28.3.1
  v2.28.9.1
  v2.21.5.1
  v2.23.4.1
  v2.25.1.1
  v2.26.3.1
  v2.26.6.1
  v2.27.5.1
  v2.27.7.1
  v2.28.7.1

Use cases

You can use this feature to diagnose issues in the following common scenarios:

• Your inference service or training job experiences exceptions, such as a task hang or high inference latency.

• Your GPU encounters an exception, such as a GPU card drop or an XID error.

• You need to perform slow node diagnosis for large model training to quickly identify slow nodes.

• The NVIDIA Collective Communications Library (NCCL) can encounter exceptions, such as a network hang or an operator hang.

Prerequisites

• If you use a RAM user, make sure that the Alibaba Cloud account (primary account) has granted the RAM user the AliyunECSReadOnlyAccess and AliyunSysomFullAccess system policies.

• The console permissions are activated.

  The first time you log on to the Operating System Console, click Activate Service to activate the console service.

• The SysOM component is installed. For installation instructions, see Component Management.

Procedure

1. Go to Operating System Console.

2. Install SysOM on the target ECS instance.

3. In the left-side navigation pane, choose GPU Performance & Diagnosis > GPU Diagnosis.

4. The following example uses slow node diagnosis. The diagnosis page consists of two sections:

   • Enter the following two diagnosis parameters:

     • Training Task ID: Typically, this is the prefix of the pod names for your training task. For example, if a two-node training task has pods named traintask-82asd33as-master-0 and traintask-82asd33as-worker-0, the Training Task ID is traintask-82asd33as.

     • Runtime duration: The diagnosis duration, in seconds.

     The value range for Runtime (s) is 0 to 60. After you enter the parameters, click Run Diagnosis.

   • Historical diagnosis results:

     • Analysis ID: The task ID generated each time you start a diagnosis.

     • Analysis Time: The time when the diagnosis was started.

     • Analysis Parameters: The details of the diagnosis parameters.

     • Analysis Status: Indicates the diagnosis status: Analyzing, Completed, or Collection Failed.

     • Actions: Provides an option to view the diagnosis report.

5. After the diagnosis is complete, go to the Analysis Records section and click View Report in the Actions column.

   • The diagnosis report for an NCCL hang issue is as follows:

     The fault diagnosis report page contains the following information: The Basic Information section displays the resource ID, diagnosis report ID, diagnosis item, and diagnosis start time. The Analysis Conclusion section lists discovered issues such as a GPU process hang and related NCCL kernel call information. The GPU Running Status section provides a table showing each GPU's fan speed, temperature, power consumption, performance state, GPU utilization, memory utilization, and running status. The GPU Process Running Status section shows each process's PID, GPU information, CPU utilization, memory, running command, and status.

   • The slow node diagnosis result page is as follows:

     The results page is divided into three parts:

     1. Basic information

        • Resource ID

        • Diagnosis Report ID

        • Diagnosis Item

        • Diagnosis Start Time

     2. Analysis Conclusion: Identifies the slow node.

     3. Raw Data: Displays the collected training task data using Perfetto.