Quick Diagnostics lets you check the network and hardware health of Lingjun nodes in PAI Lingjun Intelligent Computing Service, and run network performance tests using multiple communication libraries and communication models.
What's covered
| Diagnostic type | Checks | When to use |
|---|---|---|
| Static configuration check | System software, network, GPU configuration | Before starting a training job, or when onboarding a new node |
| Dynamic runtime check | TCP connectivity, TCP latency, Remote Direct Memory Access (RDMA) connectivity | When you suspect a live network issue during or after a job |
| System hardware diagnostics | Hardware status of Lingjun nodes | When you suspect a hardware fault |
| General network test | RDMA traffic and latency between nodes | To baseline or benchmark cluster network performance |
| Communication library test | Collective communication performance using ACCL (Alibaba Cloud Collective Communication Library) or NCCL (NVIDIA Collective Communications Library) | To verify collective communication health before distributed training |
Network diagnostics applies to instances in billing state only.
Self-service diagnostics
Network diagnostics
Static configuration check
Runs static configuration diagnostics on Lingjun nodes, covering system software, network, and GPU checks.
Log on to the Lingjun console.
In the left navigation pane, choose Quick Diagnostics > Self-service Diagnostics.
Click the Network Diagnostics tab.
Click Static Configuration Check.
In the Diagnostic Information area, select the target Cluster Name and Node ID from the drop-down lists.
Click Start Diagnostics.
Dynamic runtime check
Runs dynamic runtime diagnostics on Lingjun nodes, covering TCP connectivity, TCP latency, and RDMA connectivity.
In the left navigation pane, choose Quick Diagnostics > Self-service Diagnostics.
Click the Network Diagnostics tab.
Click Dynamic Runtime Check.
In the Diagnostic Information area, select the target Cluster Name and Node ID from the drop-down lists.
Click Start Diagnostics.
Server diagnostics
Checks the hardware status of Lingjun nodes and displays results in a visual format.
In the left navigation pane, choose Quick Diagnostics > Self-service Diagnostics.
Click the Server Diagnostics tab.
Click System Hardware Diagnostics.
In the Diagnostic Information area, select the target Cluster Name and Node ID from the drop-down lists.
Click Start Diagnostics.
Network tests
General network test
Measures RDMA traffic and latency between Lingjun nodes.
In the left navigation pane, choose Quick Diagnostics > Network Test.
Click the General Network Test tab.
In the Test Information area, select a Network Protocol and Test Type, then configure the parameters.
Click Start Diagnostics.
RDMA traffic test parameters
| Parameter | Description |
|---|---|
| Traffic Model | MtoN model: Tests one-way connectivity from Clients nodes to Servers nodes, including single-to-single and many-to-many configurations. Fullmesh model: Tests connectivity between every pair of target Lingjun nodes. |
| Test Duration | Fixed duration selected from the drop-down list. Unit: seconds. |
| QP | Number of test process streams. Affects test bandwidth. |
| GDR | When enabled, attaches Lingjun network interface controllers (NICs) to the corresponding GPUs for the traffic test. |
| Cluster Name | The cluster where the target Lingjun nodes are located. |
| Clients | (MtoN model) Select the Clients nodes. |
| Servers | (MtoN model) Select the Servers nodes. |
| Select Target Nodes | (Fullmesh model) Select the target nodes. |
| Node Port | The starting port used for the test. |
RDMA latency test parameters
| Parameter | Description |
|---|---|
| Cluster Name | The cluster where the target Lingjun nodes are located. |
| Test Nodes | Checks network latency between every pair of the selected nodes. |
| Node Port | The starting port used for the test. |
Communication library test
Tests collective communication performance using ACCL or NCCL across target Lingjun nodes.
In the left navigation pane, choose Quick Diagnostics > Network Test.
Click the Communication Library Test tab.
Configure the Test Information.
Click Start Test.
Communication library test parameters
| Parameter | Description |
|---|---|
| Communication Library Category | ACCL (Alibaba Cloud Collective Communication Library) or NCCL (NVIDIA Collective Communications Library). |
| Communication Model | The collective communication operation to test. See the table below for details. |
| Number of GPUs | The value range is 1 to 8. |
| Cluster Name | The cluster where the target Lingjun nodes are located. |
| Select Target Nodes | Specify the IP address of each target node. No need to specify the Lingjun node to start the test. |
| Node Port | The starting port used for the test. |
Communication models
| Model | What it does | Typical use case |
|---|---|---|
| AllReduce | Aggregates data from all processes, reduces it to a single value, and distributes the result back to all processes. | Gradient synchronization — the most common operation in distributed training. |
| AllGather | Gathers data from all processes into a single structure accessible by every process. | Collecting distributed shards (e.g., model parameters, embeddings). |
| AllGatherA | Extends AllGather with data type parameters, supporting large and custom data types. | Use when AllGather doesn't cover your data type requirements. |
| AllToAll | Each process distributes its data to all other processes, and receives data from all others. | Data redistribution across nodes, such as expert routing in MoE models. |
| AllToAllA | Extends AllToAll with data type and buffer parameters, supporting data of different sizes and types. | Use when communicating variable-size data across processes. |
| Broadcast | Distributes data from one process to all other processes. | Parameter initialization or broadcasting control signals. |
View diagnostic reports
The diagnostic history shows reports for all Self-service Diagnostics and Network Tests. Each report includes a Report ID and Cluster Name. Click the tabs to filter reports by diagnostic type.
For any report, the Operations column provides the following actions:
View Report: View the full results and details of the diagnostic run.
Diagnose Again: Re-run the same diagnostic.