GPU diagnosis

Limitations

• Region limitations

  This feature is available only in the Chinese mainland and China (Hong Kong).

• Operating system limitations

  Architecture	Operating system
  x86	Alibaba Cloud Linux 3 Alibaba Cloud Linux 2 Ubuntu 22.04 Ubuntu 24.04

• Instance limitations

  This feature is available only for GPU instances.

• Slow node diagnosis limitations

  Supported NCCL versions for slow node diagnosis

  v2.22.3.1
  v2.24.3.1
  v2.26.2.1
  v2.26.5.1
  v2.27.3.1
  v2.27.6.1
  v2.28.3.1
  v2.28.9.1
  v2.21.5.1
  v2.23.4.1
  v2.25.1.1
  v2.26.3.1
  v2.26.6.1
  v2.27.5.1
  v2.27.7.1
  v2.28.7.1

Use cases

You can use this feature to diagnose issues in the following common scenarios:

• Your inference service or training job experiences exceptions, such as a task hang or high inference latency.

• Your GPU encounters an exception, such as a GPU card drop or an XID error.

• You need to perform slow node diagnosis for large model training to quickly identify slow nodes.

• The NVIDIA Collective Communications Library (NCCL) can encounter exceptions, such as a network hang or an operator hang.

Prerequisites

• If you are using a RAM user, ensure that your Alibaba Cloud account (primary account) has granted the AliyunECSReadOnlyAccess and AliyunSysomFullAccess system policies to the RAM user.

• Enable the O&M console.

  The first time you log on to the O&M console, click Enable Service to activate the service.

• The SysOM component is installed. For installation instructions, see Component Management.

Procedure

1. Go to the O&M console.

2. Install SysOM on the target ECS instance.

3. In the left-side navigation pane, choose GPU Performance & Diagnosis > GPU Diagnosis.

4. The following example uses slow node diagnosis. The diagnosis page consists of two sections:

   • Enter the following two diagnosis parameters:

     • Training Task ID: Typically, this is the prefix of the pod names for your training task. For example, if a two-node training task has pods named traintask-82asd33as-master-0 and traintask-82asd33as-worker-0, the Training Task ID is traintask-82asd33as.

     • Runtime duration: The diagnosis duration, in seconds.

     The value range for Runtime (s) is 0 to 60. After you enter the parameters, click Run Diagnosis.

   • Historical diagnosis results:

     • Analysis ID: The task ID generated each time you start a diagnosis.

     • Analysis Time: The time when the diagnosis was started.

     • Analysis Parameters: The details of the diagnosis parameters.

     • Analysis Status: Indicates the diagnosis status: Analyzing, Completed, or Collection Failed.

     • Actions: Provides an option to view the diagnosis report.

5. After the diagnosis is complete, go to the Analysis Records section and click View Report in the Actions column.

   • The following is an example of a diagnosis report for an NCCL hang issue:

     The fault diagnosis report contains several sections. The Basic Information section displays the Resource ID, Diagnosis Report ID, Diagnosis Item, and Diagnosis Start Time. The Analysis Conclusion section lists discovered problems, such as a GPU process hang and related NCCL kernel call information. The GPU Running Status section provides a table with details for each GPU, including fan speed, temperature, power consumption, performance state, GPU utilization, memory utilization, and running status. The GPU Process Running Status section shows the PID, GPU information, CPU utilization, memory usage, running command, and status for each process.

   • The following is an example of a slow node diagnosis result page:

     The results page is divided into three parts:

     1. Basic information

        • Resource ID

        • Diagnosis Report ID

        • Diagnosis Item

        • Diagnosis Start Time

     2. Analysis Conclusion: Identifies the slow node.

     3. Raw Data: Displays the collected training task data using Perfetto.