How to deploy SMC-R and Redis on eRDMA-enabled enterprise-level CPU instances-Elastic Compute Service(ECS)-阿里云帮助中心

Elastic RDMA (eRDMA) provides ultra-low latency to help you process requests faster. This topic describes how to deploy shared memory communications (SMC) and Redis with eRDMA, and then benchmark their performance.

Background information

Shared memory communications (SMC) is a high-performance kernel-space protocol stack that uses shared memory and is compatible with the socket layer. SMC comes in two versions, depending on the shared memory technology used: SMC over DMA (SMC-D), based on internal shared memory (ISM), and SMC over RDMA (SMC-R), based on remote direct memory access (RDMA). For more information, see SMC applicability.
Alibaba Cloud Linux 3 uses Elastic RDMA (eRDMA) to provide SMC-R acceleration. To use SMC-R on Alibaba Cloud, you must create ECS instances that support eRDMA.

Prerequisites

Create two eRDMA-capable ECS instances. During instance creation, select the option to automatically install the eRDMA driver and enable the eRDMA interface on the primary ENI. One ECS instance will serve as the server and the other as the client. For more information, see Create an instance by using the wizard.

This topic uses the following parameters as an example:

Instance type: ecs.g8i.8xlarge
Image: Alibaba Cloud Linux 3.2104 LTS 64-bit
Network: Both instances are in the same security group and can communicate by default over their internal network.
Private IP addresses of the primary ENIs: 192.168.0.25 (server) and 192.168.0.24 (client). You must replace the sample IP addresses with your own.
Note
- This topic uses an example where the eRDMA interface is enabled on the primary ENI of an instance. In this case, 192.168.0.25 is the private IP address of the primary ENI on the server instance.
- If you enable the eRDMA interface on a secondary elastic network interface for your test, replace this IP address with the private IP address of that secondary ENI. For more information, see Step 3: Bind an ERI to an ECS instance.

Example on how to configure specific parameters during ECS instance creation

Note the following parameters when creating the instances. For other parameters, see Custom launch ECS instances.

Instances & Images: Select an instance type that supports eRDMA and install the eRDMA driver.
- Instance: For more information, see Limitations.
- Images: Select Public Image.
- Extension: Select eRDMA Driver. The eRDMA driver is automatically installed when the instance starts. When you create an Arm instance that uses an Alibaba Cloud Linux image, you can also select performance-acceleration extensions. For more information, see Application performance acceleration.
  Important
  To use the extension feature, you must have the AliyunECSExtensionsFullAccess system policy. Alibaba Cloud accounts have this permission by default. If you are a RAM user, contact an Alibaba Cloud account administrator to grant this permission to your RAM user. For more information, see Extensions.
ENIs: To the right of Primary ENI, enable the ERI feature to bind an ERI to the ECS instance.
Note
When you create an enterprise-level instance, you can enable the ERI feature only for the primary ENI. If you need to configure eRDMA for a secondary ENI, you can enable the ERI feature for the secondary ENI in the console or by calling an API operation. For more information, see Elastic RDMA network interface card (ERI).

Step 1: Deploy SMC

Remotely connect to the server and client ECS instances.
For more information, see Connect to a Linux instance by using Workbench.
On both ECS instances, run the following command to deploy the SMC-R module:
```
modprobe smc && modinfo smc
```
On both instances, run the following command to disable IPv6, which allows eRDMA connections to be established over the eRDMA interface.
Important
- Alibaba Cloud eRDMA devices and SMC do not currently support IPv6 addresses. If your application uses an IPv6 address, SMC falls back to TCP. For more information, see SMC falls back to TCP with IPv6 addresses.
- Starting from ANCK 5.10.134-17.3, SMC supports IPv4-mapped IPv6 addresses.
```
sysctl net.ipv6.conf.all.disable_ipv6=1
```
On both instances, run the following command to enable SMC for eRDMA to prioritize traffic over eRDMA.
Newly created TCP sockets are converted to SMC sockets. Existing TCP sockets are not affected.
```
sysctl net.smc.tcp2smc=1
```

On both instances, run the following command to install smc-tools, the SMC monitoring and diagnostic toolkit.

smc-tools is an IBM-developed toolkit for monitoring and diagnosing SMC-R. It offers comprehensive diagnostic and tracking tools for SMC-R, including the following:

smcr: provides statistics about SMC-R resources.
smcss: provides information about SMC sockets.

yum install -y smc-tools

After the installation is complete, you can run the smcss -a command to view the current traffic path.

[root@iZxxx]# smcss -a
State          UID   Inode  Local Address            Peer Address              Intf Mode
ACTIVE      00000 4257589 192.168.8.123:45058      192.168.8.126:20003       0000 SMCR
ACTIVE      00000 4264206 192.168.8.123:20004      192.168.8.130:54626       0000 SMCR
ACTIVE      00000 4264207 192.168.8.123:20004      192.168.8.136:54900       0000 SMCR
ACTIVE      00000 4308684 192.168.8.123:20004      192.168.8.134:38578       0000 SMCR
ACTIVE      00000 4308694 192.168.8.123:20004      192.168.8.126:60554       0000 SMCR
ACTIVE      00000 4308692 192.168.8.123:20004      192.168.8.126:60568       0000 SMCR
ACTIVE      00000 4261731 192.168.8.123:49490      192.168.8.122:20004       0000 SMCR
ACTIVE      00000 4257634 192.168.8.123:49500      192.168.8.122:20004       0000 SMCR
ACTIVE      00000 4257638 192.168.8.123:42866      192.168.8.129:20003       0000 SMCR
ACTIVE      00000 4264236 192.168.8.123:53768      192.168.8.135:20003       0000 SMCR
ACTIVE      00000 4272479 192.168.8.123:34978      192.168.8.128:20003       0000 SMCR
ACTIVE      00000 4277256 192.168.8.123:42840      192.168.8.127:20003       0000 SMCR
LISTEN      00000 4253132 0.0.0.0:38567

Step 2: Deploy Redis

After deploying SMC on the ECS instances, you must also deploy Redis.

Remotely connect to the server and client ECS instances.
For more information, see Connect to a Linux instance by using Workbench.
Run the following command to install Redis on both instances:
```
sudo yum install -y redis
```

On the server instance, run the following command to start the Redis server:

redis-server --bind 192.168.0.25 --port 6379 --protected-mode no --save

Note

In the command, 192.168.0.25 is the private IP address of the server's primary ENI, and 6379 is the listening port. Replace these with your own values.

[root@xxx Z ~]# redis-server --bind 192.168.0.25 --port 6379 --protected-mode no --save
69313:C 23 Apr 2025 15:14:05.302 # o000o000o000o Redis is starting o000o000o000o
69313:C 23 Apr 2025 15:14:05.302 # Redis version=6.2.17, bits=64, commit=00000000, modified=0, pid=69313, just started
69313:C 23 Apr 2025 15:14:05.302 # Configuration loaded
69313:M 23 Apr 2025 15:14:05.302 * monotonic clock: POSIX clock_gettime
                Redis 6.2.17 (00000000/0) 64 bit
  Running in standalone mode
  Port: 6379
  PID: 69313
                https://redis.io
69313:M 23 Apr 2025 15:14:05.303 # Server initialized
69313:M 23 Apr 2025 15:14:05.303 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
69313:M 23 Apr 2025 15:14:05.303 * Ready to accept connections

On the Redis client, connect to the server and run a test.
- Run the following command to connect to the Redis server:
```
redis-cli -h 192.168.0.25 -p 6379
```
- Run the following command to start a redis-benchmark stress test.
  The following command simulates 100 clients sending 1,000,000 SET requests to the server:
```
redis-benchmark -h 192.168.0.25 -p 6379 -n 1000000 -t set -c 100
```

For a mixed-workload stress test, you can run multiple processes after enabling SMC-R. First, deploy a second Redis client by following the steps above. Then, start the stress tests on both clients and monitor the operations per second (OPS) on the Redis server.

Example command to start eight SET stress test processes on a Redis client:

redis-benchmark -h 192.168.0.25 -p 6379 -n 100000000 -t set --threads 8 -c 100

Example command to start eight GET stress test processes on a Redis client:

redis-benchmark -h 192.168.0.25 -p 6379 -n 1000000 -t get --threads 8 -c 100

Example command to check the OPS on the Redis server:
```
redis-cli -h 192.168.0.25 -p 6379 info | grep instantaneous_ops_per_sec
```
Note
Run the command in a new remote connection window.