GPU resource topology

Prerequisites

Resource type: ACK GPU cluster

GPU card: NVIDIA GPUs

NCCL version: v2.22.23

This feature requires SysOM component version 3.9.1 or later. If your version is earlier than 3.9.1, update the SysOM component first.

Configure and enable the feature

You can enable or disable the GPU resource topology feature for a specific ACK GPU cluster by creating and applying a custom configuration template.

1. Log on to the SysOM console. In the left-side navigation pane, choose Component Management.

2. Click Create Configuration.

3. In the Create Configuration panel, set the following parameters:

   • Configuration name: Enter an easily recognizable name, such as GPU resource topology.

   • Enable GPU topology: Make sure this checkbox is selected.

     Important

     Enabling GPU resource topology increases the memory limit of the SysOM Agent by 100 MB. For example, if the default limit is 350 MB, it will be increased to 450 MB.

4. Click OK to create the configuration template.

Apply the configuration to the target cluster

This step applies the configuration template to your GPU cluster to activate the feature.

1. In the left-side navigation pane, choose Resource Management > Managed.

2. Click the Cluster tab. Find the ACK GPU cluster that you want to manage and click Change Component in the Actions column. On the Managed tab, the cluster list shows details such as the SysOM component version and SysOM component configuration for each cluster.

3. From the Configuration template drop-down list, select the template that you created, such as GPU resource topology. The Change component configuration dialog box appears, showing information about the target cluster.

4. Click Submit and wait for the component change task to complete. After the task is finished, the GPU resource topology feature is enabled on the target cluster. A dialog box appears, showing the execution status for each node.

Disable GPU resource topology

If you need to disable this feature, you can modify the configuration template and reapply it.

1. On the Component Management page, find the configuration template that you created and click Modify.

2. Clear the Enable GPU topology checkbox and then click OK. At the bottom of the configuration panel, locate the GPU topology user settings section.

3. Repeat the steps in the Apply the configuration to the target cluster section to reapply the modified template to the cluster. Once the changes take effect, the feature is disabled.

Query type options

The query type drop-down list supports three objects: AI task list, Node list, and Pod list. You can select a specific AI task, node, or Pod to view its topology. If you do not select a query type, the view displays the topology for the entire GPU cluster by default.

• AI task list: Displays all tasks in the selected cluster. You can select a specific task to view its topology.

• Node list: Displays all nodes in the selected cluster. You can select a specific node to view its topology.

• Pod list: Displays all Pods in the selected cluster. You can select a specific Pod to view its topology.

Resource topology view

After you click Query, the view displays the GPU topology for the specified object. If no object is specified, the topology for the entire GPU cluster is shown. A color-coded legend above the topology diagram explains the different states of a GPU card, such as high or low utilization, abnormal status (for example, a hang), or no monitoring data.

• A circle represents a node (machine), and the squares inside it represent the GPU cards on that node. The squares are colored based on the resource usage of each GPU card.

• The lines connecting GPU cards represent inter-card communication.

• Hover over a GPU card within a node to view its monitoring metrics, such as GPU utilization, GPU memory usage, Xid error count, NVLink bandwidth, and PCIe Tx/Rx bytes.

• Hover over a connection line to view the send (Tx) and receive (Rx) counts between GPUs. Note that for intra-node P2P connections, these counts are not supported; the line only indicates the data flow direction in collective communication. For inter-node (NET) connections, the Tx/Rx counts are supported.

  The hover information also includes basic parameters such as NCCL version, Communication, Nrank, and Channel.

• Hover over a node to view the instance ID.

Collective communication topology view

• Parameters for collective communication topology:

  • Communicator: In distributed training, a communicator (or communication group) is a set of processes or nodes that participate in a collective communication operation. It defines which devices (such as GPUs) or nodes participate in a specific communication operation.

  • Channel: A channel is a logical or physical path for communication that manages data transfer routes and resources. It is a set of communication resources (such as CUDA streams and memory buffers). Each channel can be bound to multiple GPUs and supports parallel operations across multiple streams.

  • Topology type: Describes the physical or logical connection pattern between nodes or devices (for example, tree, ring, or mesh). It is a key basis for selecting collective communication algorithms and optimizing performance.

• In Topology view options, select Collective communication topology. Then, select a communicator, channel, and topology type, and click Query. The view displays the GPU communication relationships for your selection. This is useful for analyzing multi-card communication issues in training tasks. For instance, you can check whether the communication topology matches your expectations (for example, you expect a ring topology but find a tree topology) or whether communication counts between GPUs are as expected. During a task hang, you can verify whether send and receive counts are balanced or identify performance issues by determining if a pair of GPUs has significantly lower communication counts than others in the same topology.