Deploy an eRDMA-enabled E-HPC cluster-Elastic High Performance Computing(E-HPC)-阿里云帮助中心

Background information

Using eRDMA, multi-node parallel High-Performance Computing (HPC) workloads on an E-HPC cluster (formerly E-HPC NEXT), such as climate and weather modeling, industrial simulation, and molecular dynamics, can achieve network performance comparable to on-premises clusters. This includes high bandwidth and low latency, which significantly improves the efficiency of numerical simulations. You can use RDMA within your existing network without deploying additional RDMA NICs. This approach provides seamless integration and is easy to use.

Prerequisites

To create an E-HPC cluster, go to the Create Cluster page. For more information, see Create a Standard Edition cluster.

The following table describes the sample cluster configuration.

	Parameter	Configuration
Cluster Configuration	Region	China (Shanghai)
	Network and Availability Zone	Select Availability Zone L
	Series	Standard Edition
	Deployment Mode	Public cloud cluster
	Cluster Type	SLURM
	Management node	Instance Specification: ecs.c7.xlarge with 4 vCPUs and 8 GiB of memory. Image: aliyun_2_1903_x64_20G_alibase_20240628.vhd Note The osu-benchmark installation package is built based on the Alibaba Cloud Linux 2.1903 LTS 64-bit image.
Compute Node and Queue	Queue Compute Nodes	Initial nodes: `2`.
	Inter-node interconnection	eRDMA network Note Only specific instance types support Elastic RDMA Interconnect (ERI). For more information, see Elastic RDMA (eRDMA) and Enable eRDMA on enterprise-level instances.
	Instance type Group	Instance Specification: ecs.c8ae.xlarge or other AMD instances of the same generation. Image: aliyun_2_1903_x64_20G_alibase_20240628.vhd
Shared File Storage	/home cluster mount directory	By default, a file system is mounted to the `/home` and `/opt` directories on the management node to serve as shared storage directories.
Shared File Storage	/opt mount directory
Software and Service Component	Software not Installed	erdma-installer mpich-aocc
Software and Service Component	Installable service components	Logon Node: Instance Specification: ecs.c7.xlarge with 4 vCPUs and 8 GiB of memory. Image: aliyun_2_1903_x64_20G_alibase_20240628.vhd

Create a cluster user. For more information, see user management.

Check the eRDMA environment

Verify the eRDMA configuration on the compute nodes.

Log on to the Elastic High Performance Computing console and click the target cluster.
In the left-side navigation pane, choose Nodes and Queues > Node. Select all compute nodes in the cluster and click Send Command.
Check the eRDMA network status and RDMA support on the compute nodes.
1. Send the following command to all compute nodes.
```
hpcacc erdma check
```
2. A result similar to the following indicates that the eRDMA configuration is correct.
3. If an error is returned, run the following command to repair the configuration.
```
hpcacc erdma repair
```
4. After the repair is complete, verify the eRDMA configuration again.

Run OSU-Benchmark tests

OSU-Benchmark evaluates the communication performance of High-Performance Computing (HPC) clusters and distributed systems. This topic uses the following two benchmarks to test communication performance over different network protocols (TCP vs. RDMA):

Latency test (osu_latency): Measures the one-way latency for point-to-point communication (the time from when process A sends a message to when process B receives it, excluding the return time). The test focuses on small messages from 1 byte to several kilobytes (KB). Small-message latency, a core metric for HPC system responsiveness, reflects low-level network performance (such as RDMA acceleration capabilities) and MPI library optimization. For example, in real-time simulations or machine learning parameter synchronization, low latency can significantly reduce communication overhead.
Bandwidth test (osu_bw): Measures the sustainable point-to-point bandwidth (the amount of data transferred per unit time). The test focuses on large messages from several kilobytes (KB) to several megabytes (MB). Bandwidth performance directly affects the efficiency of large data transfers, such as matrix exchange in scientific computing or file I/O scenarios. If the measured bandwidth is far below the theoretical value, you may need to optimize MPI configurations (such as multithreaded communication) or check network settings (such as MTU and flow control).

Procedure

As the user you created, connect to the E-HPC cluster. For more information, see Connect to a cluster.
Run the following command to check whether the dependent environment modules are correctly installed.
```
module avail
```

Run the following commands to download and decompress the pre-compiled OSU-Benchmark installation package.

cd ~ && wget https://ehpc-perf.oss-cn-hangzhou.aliyuncs.com/AMD-Genoa/osu-bin.tar.gz
tar -zxvf osu-bin.tar.gz

Run the following commands to go to the test directory and edit the Slurm job script.

cd ~/pt2pt 
vim slurm.job

The content of the test script is as follows:

#!/bin/bash
#SBATCH --job-name=osu-bench
#SBATCH --ntasks-per-node=1
#SBATCH --nodes=2
#SBATCH --partition=comp
#SBATCH --output=%j.out
#SBATCH --error=%j.out

# load env params
module purge
module load aocc/4.0.0 gcc/12.3.0 libfabric/1.16.0 mpich-aocc/4.0.3

# run mpi latency test: erdma
echo -e "++++++ use erdma for osu_lat: START"
mpirun -np 2 -ppn 1 -genv FI_PROVIDER="verbs;ofi_rxm" ./osu_latency
echo -e "------ use erdma for osu_lat: END\n"
# run mpi latency test: tcp
echo -e "++++++ use tcp for osu_lat: START"
mpirun -np 2 -ppn 1 -genv FI_PROVIDER="tcp;ofi_rxm" ./osu_latency
echo -e "------ use erdma for osu_lat: END\n"

# run mpi bandwidth test: erdma
echo -e "++++++ use erdma for osu_bw: START"
mpirun -np 2 -ppn 1 -genv FI_PROVIDER="verbs;ofi_rxm" ./osu_bw
echo -e "------ use erdma for osu_bw: END\n"
# run mpi bandwidth test: tcp
echo -e "++++++ use tcp for osu_bw: START"
mpirun -np 2 -ppn 1 -genv FI_PROVIDER="tcp;ofi_rxm" ./osu_bw
echo -e "------ use tcp for osu_bw: END\n"

Note

-np 2: Specifies the total number of processes. In this case, it is set to 2, which means the MPI job starts two processes.
-ppn 1: Specifies the number of processes per node. In this case, it is set to 1, which means one process runs on each node.
-genv: Sets environment variables that apply to all processes.
- FI_PROVIDER="tcp;ofi_rxm": Uses the TCP protocol and enhances communication reliability through the RXM framework.
- FI_PROVIDER="verbs;ofi_rxm": Prioritizes the high-performance Verbs protocol, which is based on RDMA, and optimizes message transmission through the RXM framework. In this topic, Alibaba Cloud eRDMA provides a high-performance elastic RDMA network.

Run the following command to submit the test job.
```
sbatch slurm.job
```
The command-line interface returns the job ID.
Run the following command to view the job status. During the test, you can also view monitoring information for the E-HPC cluster, such as storage, job, and node metrics, in the console. For more information, see View monitoring information.
```
squeue
```
In the current directory, you can view the log file that corresponds to the job ID. The output is similar to the following content:
- Network latency test results: The results show the relationship between the message size (Size in bytes, from 1 B to 4 MB) and the average network latency (Avg Latency). The following figures show the sample test results in this topic.
  
  Verbs protocol (eRDMA)
  
  TCP protocol
  
  The test data indicates that for small messages (1 B to 8 KB), eRDMA latency is significantly lower than TCP latency.
- Network bandwidth test results: The results show the relationship between the message size (Size in bytes, from 1 B to 4 MB) and the bandwidth (Bandwidth). The following figures show the sample test results in this topic.
  
  Verbs protocol (eRDMA)
  
  TCP protocol
  
  The test data indicates that for message sizes from 16 KB to 64 KB, eRDMA fully utilizes the network bandwidth, whereas the TCP protocol stack introduces additional overhead.