Enable eRDMA on enterprise-level instances

Test the eRDMA network latency.

1. On the server instance, run the following command to startib_write_lat as a server that listens for connections from the client.

   ib_write_lat -R -a -F

   • -R: uses the RDMA Connection Manager (RDMA_CM) to establish a connection.

     Important

     • For CPU-based instance types that support eRDMA, the eRDMA kernel driver is installed in Standard mode by default, which supports only the RDMA_CM connection establishment method. For more information, see Connection establishment method.

     • By default, perftest uses an out-of-band (OOB) connection. When you run a perftest on a CPU-based instance, you can use the-R parameter on both the server and the client to specify the RDMA_CM connection establishment method. Otherwise, the connection may fail.

     • You can also use the command line to make the RDMA_CM and OOB connection establishment methods compatible. For more information, see Modify the connection establishment mode of eRDMA and bRPC to achieve compatibility. After you enable compatibility, the -R parameter is no longer required.

   • -a: runs tests for all message sizes from 2 bytes to 2^23 bytes. This allows you to test the impact of different message sizes on latency.

   • -F: Forces the overwrite of any existing connections. This means that if a connection has already been established, using the -F option ignores these connections and forces a new connection to be established.

2. On the client instance, run the following command to startib_write_lat and connect to the server.

   ib_write_lat -R -a -F <server_ip>

   Replace<server_ip> with theprivate IP address of theserver's ERI-enabled ENI. To obtain the IP address, see View IP addresses.

3. View the test results.

   After the test on the client is complete,ib_write_lat outputs test configuration information, connection information, and performance test results, including latency statistics such as minimum, maximum, and average latency.

   [root@xxx              ~]# ib_write_lat -R -a -F 172.17.0.131
    RDMA_Write Latency Test
    Dual-port       : OFF		Device           : erdma_0
    Number of qps   : 1		Transport type   : IW
    Connection type  : RC		Using SRQ        : OFF
    PCIe relax order: OFF
    ibv_wr* API     : OFF
    TX depth         : 1
    Mtu              : 1024[B]
    Link type        : Ethernet
    GID index        : 0
    Max inline data  : 96[B]
    rdma_cm QPs      : ON
    Data ex. method  : rdma_cm
    local address: LID 0000 QPN 0x000b PSN 0xaf51c5
    GID: 00:22:62:22:161:180:00:00:00:00:00:00:00:00:00:00
    remote address: LID 0000 QPN 0x000a PSN 0x347ff0
    GID: 00:22:62:09:142:189:00:00:00:00:00:00:00:00:00:00
    #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec]
      2       1000          12.26          16.79          12.59            12.65            0.19              14.32                    16.79
      4       1000          12.10          18.10          12.54            12.60            0.24              14.57                    18.10
      8       1000          12.04          18.75          12.42            12.48            0.31              14.09                    18.75
     16       1000          12.45          14.77          12.77            12.81            0.11              14.10                    14.77
     32       1000          12.28          15.55          12.75            12.81            0.21              14.67                    15.55
     64       1000          12.41          16.10          12.79            12.84            0.16              14.31                    16.10
    128       1000          12.86          16.22          13.19            13.21            0.15              14.78                    16.22
    256       1000          13.11          20.85          13.53            13.64            0.35              15.43                    20.85
    512       1000          13.09          16.54          13.44            13.47            0.17              15.16                    16.54

   Description of the ib_write_lat test result fields

   • #bytes: The size of the message payload used in the test, in bytes. The size ranges from 2 bytes to 8,388,608 bytes. Different message sizes help you understand performance under different loads.

   • #iterations: The number of times the test is repeated for each message size. A higher number of iterations provides more stable averages and statistics.

   • t_min[usec]: The minimum latency recorded across all measurements, in microseconds. This value provides a reference for the best-case network latency.

   • t_max[usec]: The maximum latency recorded across all measurements, in microseconds. A high maximum latency may indicate network issues or transient congestion.

   • t_typical[usec]: The typical latency observed during the test, in microseconds. This is usually the median of all measured values.

   • t_avg[usec]: The average latency of all measurements, in microseconds. This value provides an overall view of the network latency.

   • t_stdev[usec]: The standard deviation of the latency values, in microseconds. This value indicates the degree of variation in latency. A smaller standard deviation indicates more stable latency, whereas a larger standard deviation indicates greater fluctuations.

   • 99% percentile[usec]: The 99th percentile of latency, in microseconds. This indicates that 99% of the measured values are lower than this value. This data point helps you understand latency performance in edge cases.

   • 99.9% percentile[usec]: The 99.9th percentile of latency, in microseconds. This indicates that 99.9% of the measured values are lower than this value. This data point helps you understand latency performance in extreme cases.

   Analyzing these metrics helps you understand your RDMA network's performance for optimization and troubleshooting. For example, a sudden latency increase at a specific message size may indicate that you need to check your network configuration or hardware performance. Significant latency fluctuations may require investigating network congestion or instability.