在阿里云第八代ECS实例中通过ASM启用SMC网络性能优化

前提条件

已添加集群到ASM实例。

使用限制

• 节点使用支持配置eRDMA的ECS实例。详细信息，请参见在企业级实例上配置eRDMA。

• 节点操作系统使用Alibaba Cloud Linux 3，详情参见使用操作系统Alibaba Cloud Linux 3。

• ASM实例版本为1.21及以上。关于如何升级实例，请参见升级ASM实例。

• ACK集群使用Terway网络插件，详情参见使用Terway网络插件。

• ACK集群配置API Server公网访问能力，详情参见控制集群API Server的公网访问能力。

• 当前版本中启用SMC网络性能优化与Sidecar Acceleration using eBPF功能互斥，后续版本中将解除此限制。

操作步骤

步骤一：节点环境初始化

SMC利用eRDMA网卡加速网络性能，在启用之前需要对节点进行相应的初始化准备。

1. 升级Alibaba Cloud Linux 3系统内核为5.10.134-16.3及以上。

   1. 使用uname -r查看当前内核版本，若内核版本为5.10.134-16.3及以上，则无需额外操作，跳过内核升级步骤。

      $ uname -r
      5.10.134-16.3.al8.x86_64

   2. 查看可安装内核版本

      $ sudo yum search kernel --showduplicates | grep kernel-5.10
      Last metadata expiration check: 3:01:27 ago on Tue 09 Apr 2024 07:40:15 AM CST.
      kernel-5.10.134-15.1.al8.x86_64 : The Linux kernel, based on version 5.10.134, heavily modified with backports
      kernel-5.10.134-15.2.al8.x86_64 : The Linux kernel, based on version 5.10.134, heavily modified with backports
      kernel-5.10.134-15.al8.x86_64 : The Linux kernel, based on version 5.10.134, heavily modified with backports
      kernel-5.10.134-16.1.al8.x86_64 : The Linux kernel, based on version 5.10.134, heavily modified with backports
      kernel-5.10.134-16.2.al8.x86_64 : The Linux kernel, based on version 5.10.134, heavily modified with backports
      kernel-5.10.134-16.3.al8.x86_64 : The Linux kernel, based on version 5.10.134, heavily modified with backports
      kernel-5.10.134-16.al8.x86_64 : The Linux kernel, based on version 5.10.134, heavily modified with backports
      [...]

   3. 安装最新版本内核或安装指定版本内核

      1. 安装最新版本内核：

         $ sudo yum update kernel

      2. 或安装指定内核版本，以 kernel-5.10.134-16.3.al8.x86_64 版本为例：

         $ sudo yum install kernel-5.10.134-16.3.al8.x86_64

2. 为ACK网络插件Terway配置弹性网卡白名单，防止其纳管即将添加的辅助eRDMA网卡。操作步骤请参见为弹性网卡（ENI）配置白名单。

3. 为集群内各节点创建和绑定一块辅助eRDMA网卡，具体操作参见为已有实例配置eRDMA。

   说明

   只需完成辅助eRDMA网卡的创建和绑定步骤即可。在此场景下，辅助eRDMA网卡需要和主网卡处在同一子网内。配置辅助eRDMA网卡方式请参见下一步骤。

4. 配置节点上的辅助eRDMA网卡

   1. 将下述脚本存放于节点的任意目录下。为脚本添加可执行权限：sudo chmod +x asm_erdma_eth_config.sh。

      asm_erdma_eth_config.sh

      #!/bin/bash
      
      #
      # Params
      #
      mode=
      mac=
      ipv4=
      mask=
      gateway=
      state=  # UP/DOWN
      
      #
      # Functions
      #
      function find_erdma_eth
      {
              echo "$(rdma link show | awk '{print $NF}')"
      }
      
      function get_erdma_eth_info
      {
              e=$1
              echo "Find ethernet device with erdma: $e"
      
              # UP/DOWN
              ip link show $e | grep -q "state UP" && state="UP" || state="DOWN"
      
              # MAC address
              mac=$(ip a show dev $e | grep ether | awk '{print $2}')
      
              # IPv4 address
              ipv4=$(curl --silent --show-error --connect-timeout 5 \
                      http://100.100.100.200/latest/meta-data/network/interfaces/macs/"$mac"/primary-ip-address \
                      2>&1)
              if [ $? -ne 0 ]; then
                      echo "failed to retrieve $e IPv4 address. Error: $ipv4"
                      exit 1
              fi
      
              # Mask
              mask=$(curl --silent --show-error --connect-timeout 5 \
                      http://100.100.100.200/latest/meta-data/network/interfaces/macs/"$mac"/netmask \
                      2>&1)
              if [ $? -ne 0 ]; then
                      echo "failed to retrieve $e network mask. Error: $mask"
                      exit 1
              fi
      
              # Gateway
              gateway=$(curl --silent --show-error --connect-timeout 5 \
                      http://100.100.100.200/latest/meta-data/network/interfaces/macs/"$mac"/gateway \
                      2>&1)
              if [ $? -ne 0 ]; then
                      echo "failed to retrieve $e gateway. Error: $mask"
                      exit 1
              fi
              echo "- state <$state>, IPv4 <$ipv4>, mask <$mask>, gateway <$gateway>"
      }
      
      function set_erdma_eth
      {
              local eths=()
      
              # find all eths with erdma
              eths=$(find_erdma_eth)
              if [ ${#eths[@]} -eq 0 ]; then
                      echo "Can't find ethernet device with erdma capability"
                      exit 1
              fi
      
              for e in ${eths[@]}
              do
                      if [[ $e == "eth0" ]]; then
                              echo "Skip eth0, no need to configure."
                              continue
                      fi
                      get_erdma_eth_info $e
                      echo "Config.."
      
                      # Enable
                      if [ "$state" == "DOWN" ]; then
                              ip link set $e up 1>/dev/null 2>&1
                              if ip link show $e | grep -q "state UP" ; then
                                      echo "- successed to set $e UP"
                              else
                                      echo "- failed to set $e UP"
                                      exit 1
                              fi
                      else
                              echo "- $e has been activated, nothing to do."
                      fi
      
                      # Set IP & mask
                      if ! ip addr show $e | grep -q "inet\b"; then
                              local eth0_metric=$(ip route | grep "dev eth0 proto kernel scope link" \
                                                      | awk '/metric/ {print $NF}')
                              ip addr add $ipv4/$mask dev $e metric $((eth0_metric + 1)) 1>/dev/null 2>&1
                              if [ $? -eq 0 ]; then
                                      echo "- successed to configure $e IPv4/mask and direct route"
                              else
                                      echo "- failed to configure $e IPv4/mask and direct route"
                              fi
                      else
                              echo "- $e has been configured with IPv4(s), nothing to do."
                      fi
      
                      echo "Complete all configurations of $e"
              done
      }
      
      function reset_erdma_eth
      {
              local eths=()
      
              # Find all eths with erdma
              eths=$(find_erdma_eth)
              if [ ${#eths[@]} -eq 0 ]; then
                      echo "Can't find ethernet device with erdma capability"
                      exit 1
              fi
      
              for e in ${eths[@]}
              do
                      if [[ $e == "eth0" ]]; then
                              echo "Skip eth0, no need to configure."
                              continue
                      fi
                      get_erdma_eth_info $e
                      echo "Reset.."
      
                      # Remove IPv4
                      ip addr flush dev $e scope global 1>/dev/null 2>&1
                      if [ $? -eq 0 ]; then
                              echo "- successed to flush $e IPv4(s)"
                      else
                              echo "- failed to flush $e IPv4(s)"
                      fi
      
                      # Disable
                      ip link set $e down 1>/dev/null 2>&1
                      if [ $? -eq 0 ]; then
                              echo "- successed to set $e DOWN"
                      else
                              echo "- failed to set $e DOWN"
                      fi
                      echo "Complete all resets of $e"
              done
      }
      
      print_help() {
              echo "Usage: $0 [option]"
              echo "Options:"
              echo "  -s            Enable eRDMA-cap Eth and configure its IPv4"
              echo "  -r            Disable eRDMA-cap Eth and remove all its IPv4"
              echo "  -h, --help    Show this help message"
      }
      
      while [ "$1" != "" ]; do
              case $1 in
                      -s)
                              set_erdma_eth
                              exit 0
                              ;;
                      -r)
                              reset_erdma_eth
                              exit 0
                              ;;
                      -h | --help)
                              print_help
                              exit 0
                              ;;
                      *)
                              echo "Invalid option: $1"
                              print_help
                              exit 1
                              ;;
              esac
              shift
      done
      
      if [ -z "$1" ]; then
              print_help
              exit 1
      fi

      说明

      此脚本仅在当前上下文中用于辅助eRDMA网卡配置，不适用于其他网卡配置场景。

   2. 执行sudo ./asm_erdma_eth_config.sh -s将新添加的辅助eRDMA网卡状态设置为UP，并为其配置IPv4地址。预计输出类似如下内容：

      $ sudo ./asm_erdma_eth_config.sh -s
      Find ethernet device with erdma: eth2
      - state <DOWN>, IPv4 <192.168.10.96>, mask <255.255.255.0>, gateway <192.168.10.253>
      Config..
      - successed to set eth2 UP
      - successed to configure eth2 IPv4/mask and direct route
      Complete all configurations of eth2

   3. （可选）上述配置辅助eRDMA网卡的步骤在每次重启节点后需要再次执行。若希望重启节点时自动执行配置，可按如下方式创建对应的systemd service。

      1. 在节点/etc/systemd/system目录下添加如下asm_erdma_eth_config.service文件，将其中的/path/to/asm_erdma_eth_config.sh更换为节点上asm_erdma_eth_config.sh脚本的实际路径。

         asm_erdma_eth_config.service

         [Unit]
         Description=Run asm_erdma_eth_config.sh script after network is up
         Wants=network-online.target
         After=network-online.target
         
         [Service]
         Type=oneshot
         ExecStart=/bin/sh /path/to/asm_erdma_eth_config.sh -s
         RemainAfterExit=yes
         
         [Install]
         WantedBy=multi-user.target

      2. 启用asm_erdma_eth_config.service。

         sudo systemctl daemon-reload
         sudo systemctl enable asm_erdma_eth_config.service

         此后在节点启动时将自动执行网卡配置。节点启动后可通过sudo systemctl status asm_erdma_eth_config.service查看asm_erdma_eth_config.service状态，通过sudo journalctl -u asm_erdma_eth_config.service查看asm_erdma_eth_config.service的打印输出。

      3. 相反，若不再需要asm_erdma_eth_config.service，可通过sudo systemctl disable asm_erdma_eth_config.service移除。

5. 执行以下命令，安装bpftool。

   $ sudo yum install bpftool
   # 检查安装结果。
   $ bpftool --version

步骤二：部署测试应用

1. 为测试使用的default命名空间启用自动注入，具体请参见启用自动注入。

2. 使用以下内容，创建fortioserver.yaml文件。

   展开查看fortioserver.yaml

   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: fortioserver
   spec:
     ports:
     - name: http-echo
       port: 8080
       protocol: TCP
     - name: tcp-echoa
       port: 8078
       protocol: TCP
     - name: grpc-ping
       port: 8079
       protocol: TCP
     selector:
       app: fortioserver
     type: ClusterIP
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: fortioserver
     name: fortioserver
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: fortioserver
     template:
       metadata:
         labels:
           app: fortioserver
         annotations:
           sidecar.istio.io/inject: "true"
           sidecar.istio.io/proxyCPULimit: 2000m
           proxy.istio.io/config: |
             concurrency: 2 
       spec:
         shareProcessNamespace: true
         containers:
         - name: captured
           image: fortio/fortio:latest_release
           ports:
           - containerPort: 8080
             protocol: TCP
           - containerPort: 8078
             protocol: TCP
           - containerPort: 8079
             protocol: TCP
         - name: anolis
           securityContext:
             runAsUser: 0
           image: openanolis/anolisos:latest
           args:
           - /bin/sleep
           - 3650d
   ---
   apiVersion: v1
   kind: Service
   metadata:
     annotations:
         service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "off"
     name: fortioclient
   spec:
     ports:
     - name: http-report
       port: 8080
       protocol: TCP
     selector:
       app: fortioclient
     type: LoadBalancer
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: fortioclient
     name: fortioclient
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: fortioclient
     template:
       metadata:
         annotations:
           sidecar.istio.io/inject: "true"
           sidecar.istio.io/proxyCPULimit: 4000m
           proxy.istio.io/config: |
              concurrency: 4
         labels:
           app: fortioclient
       spec:
         shareProcessNamespace: true
         affinity:
           podAntiAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
             - labelSelector:
                 matchExpressions:
                 - key: app
                   operator: In
                   values:
                   - fortioserver
               topologyKey: "kubernetes.io/hostname"
         containers:
         - name: captured
           volumeMounts:
           - name: shared-data
             mountPath: /var/lib/fortio
           image: fortio/fortio:latest_release
           ports:
           - containerPort: 8080
             protocol: TCP
         - name: anolis
           securityContext:
             runAsUser: 0
           image: openanolis/anolisos:latest
           args:
           - /bin/sleep
           - 3650d
         volumes:
         - name: shared-data
           emptyDir: {}
   

3. 使用ACK集群的KubeConfig，执行以下命令，部署测试应用。

   kubectl apply -f fortioserver.yaml

4. 执行以下命令，查看测试应用的状态。

   kubectl get pods | grep fortio

   预期输出：

   NAME                            READY   STATUS    RESTARTS      
   fortioclient-8569b98544-9qqbj   3/3     Running   0
   fortioserver-7cd5c46c49-mwbtq   3/3     Running   0

   预期输出表明两个应用均正常启动。

步骤三：在基础环境运行测试，查看基线测试结果

fortio应用启动后，会暴露8080端口监听，访问该端口将打开fortio应用的控制台页面。为了生成测试流量，可以将fortioclient的端口映射到当前所用机器，在当前所用机器上打开fortio的控制台页面。

1. 使用ACK集群的KubeConfig，执行以下命令，将fortio客户端的Service监听的8080端口映射到本地的8080端口。

   kubectl port-forward service/fortioclient 8080:8080

2. 在浏览器中输入http://localhost:8080/fortio地址，访问fortio客户端控制台，并修改相关配置。

   

   请按照下表修改页面上的参数。

   参数	示例值
   URL	http://fortioserver:8080/echo
   QPS	100000
   Duration	30s
   Threads/Simultaneous connections	64
   Payload	填写以下字符串（128 Byte）： xhsyL4ELNoUUbC3WEyvaz0qoHcNYUh0j2YHJTpltJueyXlSgf7xkGqc5RcSJBtqUENNjVHNnGXmoMyILWsrZL1O2uordH6nLE7fY6h5TfTJCZtff3Wib8YgzASha8T8g

3. 配置完成后，在页面下方，单击Start开始测试，等待进度条结束，测试完毕。

   

   测试运行完毕后，页面将给出本次测试的结果。下图仅供参考，测试结果请以实际环境为准。

   

   页面输出的测试结果横坐标为请求的Latency，观察柱形图在横坐标上的分布可以得出请求延迟的分布情况，紫色曲线为在不同响应时间范围内完成的请求数量。纵坐标为完成的请求数。同时，图表顶部给出了P50/P75/P90/P99/P99.9的请求Latency数据。得到基础环境数据后，需要为应用启用SMC，准备进行SMC加速后的性能验证。

步骤四：为ASM实例和工作负载启用SMC加速

1. 使用服务网格的KubeConfig编辑网格配置，添加"smcEnabled: true"，以启用SMC加速功能。

   $ kubectl edit asmmeshconfig
   
   apiVersion: istio.alibabacloud.com/v1beta1
   kind: ASMMeshConfig
   metadata:
     name: default
   spec:
     ambientConfiguration:
       redirectMode: ""
       waypoint: {}
       ztunnel: {}
     cniConfiguration:
       enabled: true
       repair: {}
     smcEnabled: true

2. 使用ACK集群的KubeConfig，执行以下命令，修改fortioserver和fortioclient的Deployment，为Pod添加Annotation。

   为网格实例启用加速后，还需要进一步为工作负载启用加速，通过为Pod添加Key为smc.asm.alibabacloud.com/enabled，值设置为true的Annotation，可以为工作负载启用SMC加速，您需要同时为需要优化的两端工作负载均启用加速。

   1. 编辑fortioclient的Deployment定义。

      $ kubectl edit deployment fortioclient
      
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        ......
        name: fortioclient
      spec:
        ......
        template:
          metadata:
            ......
            annotations:
              smc.asm.alibabacloud.com/enabled: "true"
              

   2. 编辑fortioserver的Deployment定义。

      $ kubectl edit deployment fortioserver
      
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        ......
        name: fortioserver
      spec:
        ......
        template:
          metadata:
            ......
            annotations:
              smc.asm.alibabacloud.com/enabled: "true"
              

步骤五：加速后环境运行测试，查看启用优化后的测试结果

由于修改Deployment将使工作负载重启，因此您需要参考步骤二重新进行fortioclient端口映射，再次发起测试，等待测试结束查看结果。

与加速前的结果对比，可以看到启用ASM SMC加速后，Latency下降，QPS明显提升。