Shared Memory Communications (SMC) diagnostics

Prerequisites

Before you begin, ensure that you have:

• Alibaba Cloud Linux 3 installed on your instance

• smc-tools installed (the user-mode SMC maintenance toolset)

To install smc-tools, run:

sudo yum install -y smc-tools

Fallback diagnostics

The SMC protocol stack automatically negotiates with the peer during connection establishment to decide whether to use SMC. If the conditions for SMC are not met, the connection safely falls back to TCP. You can use the smcss command to view the fallback reason for each SMC connection. For more information, see Connection monitoring.

Packet capture diagnostics

The SMC protocol stack uses technologies such as elastic Remote Direct Memory Access (eRDMA) for data transmission. This process bypasses the traditional network protocol stack. Therefore, traditional sniffing points cannot directly capture data and control packets.

Starting from version ANCK 5.10.134-18, the Alibaba Cloud Linux 3 kernel provides a solution to capture data and control packets from the SMC protocol stack:

• When the SMC kernel stack sends or receives connection control messages and data blocks in shared memory, it constructs a UDP socket buffer (skb). The skb points to the target data memory and is sent to a specific virtual network device (dummy device). The virtual network device silently consumes the skb, which never actually enters the network. During this process, the skb passes through existing kernel sniffing points, such as dev_queue_xmit_nit(). This allows tcpdump and libpcap to capture the data and control packets that the SMC stack sends or receives on the virtual network device.

  

• The TCP network packets from the SMC handshake process are still captured on the original Ethernet network interface controller (NIC).

The following table summarizes the packet capture methods for these two stages.

smc-tools provides the smcdump-ex packet capture tool based on this solution. smcdump-ex is a wrapper script for the tcpdump tool that captures the entire SMC communication process. The script creates a virtual network device named smc-dummy{four-random-characters} in the current net namespace and enables the capture of SMC data and control packets. It uses tcpdump to capture SMC handshake negotiation packets, data packets, and control packets on Ethernet or virtual network devices as required. When it receives a SIGINT signal (Ctrl+C), it stops the capture, disables the capture feature, and destroys the virtual network device.

• Tool usage

  You can run the following command to view the usage of smcdump-ex.

  Warning

  smcdump-ex is an experimental tool. Its usage may change in the future.

  smcdump-ex -h
  usage: smcdump-ex [-h] [-m {all,smc,smcd,smcr}] [-t {all,raw,cdc}]
                    [--param PARAM] [--filter FILTER] [--legacy]
  SMC Dump - SMC Traffic Capture (Experimental)
  optional arguments:
    -h, --help            show this help message and exit
    -m {all,smc,smcd,smcr}, --mode {all,smc,smcd,smcr}
                          Select the mode (default: smc)
    -t {all,raw,cdc}, --type {all,raw,cdc}
                          Select the packet type (default: all)
    --param PARAM         Additional parameters for tcpdump. e.g. --param '-w
                          packets.pcap'
    --filter FILTER       Additional filter expressions for tcpdump. e.g.
                          --filter 'host xxx.xxx.x.x and port 8080'
    --legacy              Use the legacy SMC dump header format

  • Use -m to set the data capture mode

    The following capture modes are available:

    • all: Captures packets on all network interfaces, which is equivalent to calling tcpdump -i any. This captures SMC handshake negotiation packets on the Ethernet interface and SMC data and control packets on the new virtual network interface. In this mode, you can use --filter to set additional tcpdump filter expressions. This narrows the scope of packet capture to accurately capture SMC negotiation packets. For more information, see the description of --filter.

    • smc, smcr, and smcd: Captures only SMC, SMC-R, or SMC-D data and control packets on the new virtual network interface. This is equivalent to calling tcpdump -i smc-dummy{four-random-characters}.

  • Use -t to set the type of network packets to capture

    The following network packet types can be captured:

    • all: SMC data and control packets.

    • raw: SMC data packets only.

    • cdc: SMC control packets only.

  • Use --param to set other tcpdump parameters

    For example, you can use --param '-w smcdata.pcap' to dump the captured content to a file.

  • Use --filter to set other tcpdump filter rules

    For example, you can use --filter 'host <ip> and port <port>' to set the IP address and port of the packets to capture. This ensures that SMC negotiation packets are accurately captured and prevents other TCP network packets from interfering with the analysis.

  • Use --legacy to parse the old smc dump header format

    This parameter is only for compatibility with kernel version ANCK 5.10.134-17.3. For kernel versions ANCK 5.10.134-18 and later, you do not need to set this parameter.

• Usage example

  For example, to capture all packets during the communication of an SMC connection with the IP address 192.168.2.5 and port 5201, and dump them to the smc.pcap file, run the following command:

  smcdump-ex -m all -t all --param '-w smc.pcap' --filter 'host 192.168.2.5 and port 5201'

  The captured results saved to a .pcap file using the --param '-w <pcap file>' parameter can be further analyzed with the Wireshark tool and a Lua plugin:

  1. Download and install Wireshark.

  2. You can use Wireshark with the Lua plugin to analyze the captured results stored in the .pcap file.

     1. Download the Lua plugin: https://os-smc-new.oss-cn-hangzhou.aliyuncs.com/smc_dump.lua.

        Note

        To use this Lua plugin, ensure that the Lua interpreter in Wireshark is version 5.3 or later.

     2. Find the installation path for the Wireshark Lua script and place the Lua script in that path.

        • You can view the installation path of the Wireshark Lua script on macOS.

          1. Run Wireshark.

          2. From the menu bar, click Wireshark > About Wireshark > Folders.

             In the list on the Folders tab, find the Global Lua Plugins row. The Location column shows the installation path of the Lua script, for example /Applications/Wireshark.app/Contents/PlugIns/wireshark.

        • You can find the installation path of the Wireshark Lua script on Windows.

          1. Run Wireshark.

          2. From the main Wireshark interface, click Help > Folders.

             Name                    Location                                    Typical files
             Global Extcap path      C:\Program Files\Wireshark\extcap           external capture (extcap) plugins
             MIB/PIB path                                                        SMI MIB/PIB search path
             MaxMind DB path         C:\ProgramData\GeoIP                        MaxMind DB database search path
             MaxMind DB path         C:\GeoIP                                    MaxMind DB database search path
             Personal Extcap path    C:\Users\xxx\App...oaming\Wireshark\extcap  external capture (extcap) plugins
             "File" dialogs          C:\Users\xxx\Documents                      Capture files
             Personal Lua Plugins    C:\Users\xxx\App...aming\Wireshark\plugins  Lua scripts
             Personal Plugins        C:\Users\xxx\App...g\Wireshark\plugins\4.2  Binary plugins
             Personal configuration  C:\Users\xxx\AppData\Roaming\Wireshark      dfilters, preferences, ethers, ...
             Temporary               C:\Users\xxx\AppData\Local\Temp             Untitled capture files
             Global Lua Plugins      C:\Program Files\Wireshark\plugins          Lua scripts          ← Place the script in this directory
             Global Plugins          C:\Program Files\Wireshark\plugins\4.2      Binary plugins
             Global configuration    C:\Program Files\Wireshark                  dfilters, preferences, manuf, ...
             Program                 C:\Program Files\Wireshark                  Program files
             System                  C:\Program Files\Wireshark                  ethers, ipxnets

  3. Restart Wireshark or reload the Lua plugins to apply the script. After this, all UDP packets that have an SMC dump header in their payload will be parsed as the SMC protocol, as shown in the following output.

     The Wireshark filter is tcp.port==36508 or udp.port==36508, and the capture shows the complete SMC-R communication process. The TCP packets (No. 20-22, 24, 26, 30, 49, 61, and 62) are handshake negotiation packets. The SMC and SMCv2 packets (No. 23, 25, 27-29, 31-33, 37-42, 45-48, and 60) are RDMA data packets and control packets that the kernel converts into UDP packets for capture.

     tcp.port==36508 or udp.port==36508
     
     No.  Time      Source  Destination  Protocol  Length  Info
     20   0.000722  xxx     xxx          TCP       84      36508 → 5201 [SYN] Seq=0 Win=59220 Len=0 MSS=8460 SACK_PERM TSval=2524508823 TSecr=0 WS=128 ExID=E2D4
     21   0.000834  xxx     xxx          TCP       84      5201 → 36508 [SYN, ACK] Seq=0 Ack=1 Win=59136 Len=0 MSS=8460 SACK_PERM TSval=4260635458 TSecr=2524508823 WS=128 ExID=E2D4
     22   0.000838  xxx     xxx          TCP       68      36508 → 5201 [ACK] Seq=1 Ack=1 Win=59264 Len=0 TSval=2524508823 TSecr=4260635458
     23   0.000842  xxx     xxx          SMCv2     224     [SMC-Rv2-Proposal],36508 → 5201 [PSH, ACK] Seq=1 Ack=1 Win=59264 Len=156 TSval=2524508823 TSecr=4260635458
     24   0.000904  xxx     xxx          TCP       68      5201 → 36508 [ACK] Seq=1 Ack=157 Win=59008 Len=0 TSval=4260635458 TSecr=2524508823
     25   0.000947  xxx     xxx          SMCv2     176     [SMC-Rv2-Accept],5201 → 36508 [PSH, ACK] Seq=1 Ack=157 Win=59008 Len=108 TSval=4260635458 TSecr=2524508823 OP=0x000aa5
     26   0.000948  xxx     xxx          TCP       68      36508 → 5201 [ACK] Seq=157 Ack=109 Win=59264 Len=0 TSval=2524508823 TSecr=4260635458
     27   0.000991  xxx     xxx          SMCv2     176     [SMC-Rv2-Confirm],36508 → 5201 [PSH, ACK] Seq=157 Ack=109 Win=59264 Len=108 TSval=2524508823 TSecr=4260635458 OP=0x0011fc
     28   0.000993  xxx     xxx          SMC       93      [Raw Data], 36508 → 5201 size=37
     29   0.000993  xxx     xxx          SMC       104     [CDC messages], 36508 → 5201 Prod=0:37 Cons=0:0
     30   0.001050  xxx     xxx          TCP       68      5201 → 36508 [ACK] Seq=109 Ack=265 Win=59008 Len=0 TSval=4260635458 TSecr=2524508823
     31   0.001156  xxx     xxx          SMC       65551   [Raw Data], 36508 → 5201 size=65495
     32   0.001336  xxx     xxx          SMC       65551   [Raw Data], 36508 → 5201 size=65495
     33   0.001543  xxx     xxx          SMC       65551   [Raw Data], 36508 → 5201 size=65495
     37   0.001548  xxx     xxx          SMC       138     [Raw Data], 36508 → 5201 size=82
     38   0.001549  xxx     xxx          SMC       104     [CDC messages], 36508 → 5201 Prod=0:131109 Cons=0:0
     39   0.001560  xxx     xxx          SMC       65551   [Raw Data], 36508 → 5201 size=65495
     40   0.001566  xxx     xxx          SMC       65551   [Raw Data], 36508 → 5201 size=65495
     41   0.001574  xxx     xxx          SMC       101     [Raw Data], 36508 → 5201 size=45
     42   0.001574  xxx     xxx          SMC       104     [CDC messages], 36508 → 5201 Prod=1:0 Cons=0:0
     45   0.001609  xxx     xxx          SMC       104     [CDC messages], 5201 → 36508 Prod=0:0 Cons=0:131109
     46   0.001610  xxx     xxx          SMC       93      [Raw Data], 36508 → 5201 size=37
     47   0.001611  xxx     xxx          SMC       104     [CDC messages], 36508 → 5201 Prod=1:37 Cons=0:0
     48   0.001649  xxx     xxx          SMC       104     [CDC messages], 36508 → 5201 Prod=0:0 Cons=0:131109
     49   0.001673  xxx     xxx          TCP       68      5201 → 36508 [FIN, ACK] Seq=109 Ack=265 Win=59008 Len=0 TSval=4260635459 TSecr=2524508823
     60   0.001835  xxx     xxx          SMC       104     [CDC messages], 36508 → 5201 Prod=1:37 Cons=0:0
     61   0.001838  xxx     xxx          TCP       68      36508 → 5201 [FIN, ACK] Seq=265 Ack=110 Win=59264 Len=0 TSval=2524508824 TSecr=4260635459
     62   0.001904  xxx     xxx          TCP       68      5201 → 36508 [ACK] Seq=110 Ack=266 Win=59008 Len=0 TSval=4260635459 TSecr=2524508824