服务网格ASM可以为运行其上的微服务提供无侵入式的流量治理能力,但是仅通过服务网格ASM无法实现无侵入全链路AB测试。引入WebAssembly(WASM)后,无需修改微服务代码,即可实现无侵入全链路AB测试。本文介绍如何基于ASM实现无侵入全链路AB测试。

前提条件

背景信息

WebAssembly(WASM)是一种有效的可移植二进制指令格式,可以用于扩展ASM数据平面的功能。关于无侵入全链路AB测试和WASM开发的更多信息,请参见基于WASM的无侵入式全链路A/B Test实践
说明 本文的镜像仓库仅供参考,请根据镜像脚本自行构建和推送镜像至自建仓库。关于镜像脚本的具体信息,请参见hello-servicemesh-grpc

步骤一:启用使用WASM Filter的功能

  1. 使用以下内容,创建名为runtime-config.json的文件。
    {
      "type": "envoy_proxy",
      "abiVersions": [
        "v0-541b2c1155fffb15ccde92b8324f3e38f7339ba6",
        "v0-097b7f2e4cc1fb490cc1943d0d633655ac3c522f",
        "v0-4689a30309abf31aee9ae36e73d34b1bb182685f",
        "v0.2.1"
      ],
      "config": {
        "rootIds": [
          "propaganda_filter_root"
        ]
      }
    }
  2. 执行以下命令,上传WASM Filter到ACR仓库。
    oras push ${WASM_REGISTRY}/propagate_header:0.0.1 \
      --manifest-config \
      --runtime-config.json:application/vnd.module.wasm.config.v1+json \
      ${WASM_IMAGE}:application/vnd.module.wasm.content.layer.v1+wasm
    • WASM_REGISTRY:容器镜像仓库地址。
    • WASM_IMAGE:当前路径下的WASM文件名。
    • runtime-config.json:当前路径下的运行时配置文件。
  3. 启用使用WASM Filter的功能。
    1. 执行以下命令,确认Alinyun CLI的版本。
      Aliyun CLI必须为3.0.73及以上版本。
      aliyun version
    2. 执行以下命令,启用使用WASM Filter的功能。
      aliyun servicemesh UpdateMeshFeature --ServiceMeshId=xxxxxx --WebAssemblyFilterEnabled=true
  4. 执行以下命令,验证服务网格实例的开启状态。
    aliyun servicemesh DescribeServiceMeshDetail \
      --ServiceMeshId $MESH_ID |
      jq '.ServiceMesh.Spec.MeshConfig.WebAssemblyFilterDeployment'

    预期输出:

    {
      "Enabled": true
    }
  5. 执行以下命令,验证asmwasm-cache的状态。
    WASM Filter功能开启后,会在ACK集群的每个节点上创建一个名为asmwasm-cache的DaemonSet。
    kubectl get daemonset -n istio-system

    预期输出:

    NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
    asmwasm-cache   4         4         4       4            4           kubernetes.io/os=linux   34

步骤二:部署实验资源

  1. 使用以下内容,在kube目录下创建名为hello.yaml的文件。
    该YAML文件包含v1和v2版本的Hello1、Hello2和Hello3应用。
    说明 您也可以在GitHub上获取Hello应用的YAML文件。详细信息,请参见kube
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hello1-deploy-v1
      labels:
        app: hello1-deploy-v1
        service: hello1-deploy
        version: v1
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: hello1-deploy-v1
          service: hello1-deploy
          version: v1
      template:
        metadata:
          labels:
            app: hello1-deploy-v1
            service: hello1-deploy
            version: v1
        spec:
          serviceAccountName: http-hello-sa
          containers:
            - name: hello-v1-deploy
              image: registry.cn-beijing.aliyuncs.com/asm_repo/http_springboot_v1:1.0.0
              env:
                - name: HTTP_HELLO_BACKEND
                  value: "hello2-svc"
              ports:
                - containerPort: 8001
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hello1-deploy-v2
      labels:
        app: hello1-deploy-v2
        service: hello1-deploy
        version: v2
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: hello1-deploy-v2
          service: hello1-deploy
          version: v2
      template:
        metadata:
          labels:
            app: hello1-deploy-v2
            service: hello1-deploy
            version: v2
        spec:
          serviceAccountName: http-hello-sa
          containers:
            - name: hello-v2-deploy
              image: registry.cn-beijing.aliyuncs.com/asm_repo/http_springboot_v2:1.0.0
              env:
                - name: HTTP_HELLO_BACKEND
                  value: "hello2-svc"
              ports:
                - containerPort: 8001apiVersion: v1
    ---
    kind: Service
    metadata:
      name: hello1-svc
      labels:
        app: hello1-svc
    spec:
      ports:
        - port: 8001
          name: http
      selector:
        service: hello1-deployapiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hello2-deploy-v1
      labels:
        app: hello2-deploy-v1
        service: hello2-deploy
        version: v1
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: hello2-deploy-v1
          service: hello2-deploy
          version: v1
      template:
        metadata:
          labels:
            app: hello2-deploy-v1
            service: hello2-deploy
            version: v1
        spec:
          serviceAccountName: http-hello-sa
          containers:
            - name: hello-v1-deploy
              image: registry.cn-beijing.aliyuncs.com/asm_repo/http_springboot_v1:1.0.0
              env:
                - name: HTTP_HELLO_BACKEND
                  value: "hello3-svc"
              ports:
                - containerPort: 8001
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hello2-deploy-v2
      labels:
        app: hello2-deploy-v2
        service: hello2-deploy
        version: v2
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: hello2-deploy-v2
          service: hello2-deploy
          version: v2
      template:
        metadata:
          labels:
            app: hello2-deploy-v2
            service: hello2-deploy
            version: v2
        spec:
          serviceAccountName: http-hello-sa
          containers:
            - name: hello-v2-deploy
              image: registry.cn-beijing.aliyuncs.com/asm_repo/http_springboot_v2:1.0.0
              env:
                - name: HTTP_HELLO_BACKEND
                  value: "hello3-svc"
              ports:
                - containerPort: 8001apiVersion: v1
    ---
    kind: Service
    metadata:
      name: hello2-svc
      labels:
        app: hello2-svc
    spec:
      ports:
        - port: 8001
          name: http
      selector:
        service: hello2-deployapiVersion: apps/v1
    ---
    kind: Deployment
    metadata:
      name: hello3-deploy-v1
      labels:
        app: hello3-deploy-v1
        service: hello3-deploy
        version: v1
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: hello3-deploy-v1
          service: hello3-deploy
          version: v1
      template:
        metadata:
          labels:
            app: hello3-deploy-v1
            service: hello3-deploy
            version: v1
        spec:
          serviceAccountName: http-hello-sa
          containers:
            - name: hello-v1-deploy
              image: registry.cn-beijing.aliyuncs.com/asm_repo/http_springboot_v1:1.0.0
              ports:
                - containerPort: 8001
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hello3-deploy-v2
      labels:
        app: hello3-deploy-v2
        service: hello3-deploy
        version: v2
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: hello3-deploy-v2
          service: hello3-deploy
          version: v2
      template:
        metadata:
          labels:
            app: hello3-deploy-v2
            service: hello3-deploy
            version: v2
        spec:
          serviceAccountName: http-hello-sa
          containers:
            - name: hello-v2-deploy
              image: registry.cn-beijing.aliyuncs.com/asm_repo/http_springboot_v2:1.0.0
              ports:
                - containerPort: 8001apiVersion: v1
    ---
    kind: Service
    metadata:
      name: hello3-svc
      labels:
        app: hello3-svc
    spec:
      ports:
        - port: 8001
          name: http
      selector:
        service: hello3-deployapiVersion: v1
    ---
    kind: ServiceAccount
    metadata:
      name: http-hello-sa
      labels:
        account: http-hello-deploy
  2. 使用以下内容,在mesh目录下创建名为mesh.yaml的文件。
    说明 您也可以在GitHub上获取Ingress Gateway、DestinationRule和VirtualService的YAML文件。详细信息,请参见mesh

    YAML文件中包含DestinationRule、VirtualService和Ingress Gateway。

    在DestinationRule设置以下子集:
    • 设置v1版本的Hello1应用为hello1v1,v2版本的Hello1应用为hello1v2。
    • 设置v1版本的Hello2应用为hello2v1,v2版本的Hello2应用为hello2v2。
    • 设置v1版本的Hello3应用为hello3v1,v2版本的Hello3应用为hello3v2。
    在VirtualService中设置了以下路由规则:
    • 仅包含route-v且版本为v2的请求才能路由到hello1v2,否则路由到hello1v1。
    • 仅包含route-v且版本为hello2v2的请求才能路由到hello2v2,否则路由到hello2v1。
    • 仅包含route-v且版本为hello3v2的请求才能路由到hello3v2,否则路由到hello3v1。
    apiVersion: networking.istio.io/v1alpha3
    kind: DestinationRule
    metadata:
      name: hello1-dr
    spec:
      host: hello1-svc
      subsets:
        - name: hello1v1
          labels:
            version: v1
        - name: hello1v2
          labels:
            version: v2
    ---
    apiVersion: networking.istio.io/v1alpha3
    kind: Gateway
    metadata:
      name: hello-gateway
    spec:
      selector:
        istio: ingressgateway
      servers:
        - port:
            number: 8001
            name: http
            protocol: HTTP
          hosts:
            - "*"
    ---
    # https://istio.io/latest/docs/reference/config/networking/virtual-service/
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: hello1-vs
    spec:
      hosts:
        - "*"
      gateways:
        - hello-gateway
      #  - mesh
      http:
        - name: hello1-v1-route
          match:
            - headers:
                route-v:
                  exact: v2
          route:
            - destination:
                host: hello1-svc
                subset: hello1v2
        - route:
            - destination:
                host: hello1-svc
                subset: hello1v1
    ---
    apiVersion: networking.istio.io/v1alpha3
    kind: DestinationRule
    metadata:
      name: hello2-dr
    spec:
      host: hello2-svc
      subsets:
        - name: hello2v1
          labels:
            version: v1
        - name: hello2v2
          labels:
            version: v2
    ---
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: hello2-vs
    spec:
      hosts:
        - hello2-svc
      http:
      - name: hello2-v2-route
        match:
        - headers:
            route-v:
              exact: hello2v2
        route:
        - destination:
            host: hello2-svc
            subset: hello2v2
      - route:
        - destination:
            host: hello2-svc
            subset: hello2v1
    ---
    apiVersion: networking.istio.io/v1alpha3
    kind: DestinationRule
    metadata:
      name: hello3-dr
    spec:
      host: hello3-svc
      subsets:
        - name: hello3v1
          labels:
            version: v1
        - name: hello3v1
          labels:
            version: v2
        - name: hello3v2
          labels:
            version: v2
    ---
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: hello3-vs
    spec:
      hosts:
      - hello3-svc
      http:
      - match:
        - headers:
            route-v:
              exact: hello3v2
        route:
        - destination:
            host: hello3-svc
            subset: hello3v2
      - route:
        - destination:
            host: hello3-svc
            subset: hello3v1
  3. 执行以下命令,部署Hello应用、Ingress Gateway、VirtualService和DestinationRule。
    alias k="kubectl --kubeconfig $USER_CONFIG"
    alias m="kubectl --kubeconfig $MESH_CONFIG"
    
    k -n "$NS" apply -f kube/kube.yaml
    m -n "$NS" apply -f mesh/mesh.yaml

步骤三:部署自定义资源ASMFilterDeployment

  1. 在ACK集群中创建用来访问镜像仓库的Secret。
    关于Secret的详细信息,请参见Secret
    1. 使用以下内容,创建名为myconfig.json的文件。
      {
          "auths":{
              "**********.cn-hangzhou.cr.aliyuncs.com":{
                  "username":"*****username*****",
                  "password":"*****password*****"
              }
          }
      }
      • **********.cn-hangzhou.cr.aliyuncs.com:镜像仓库地址。
      • username:镜像仓库用户名。
      • password:镜像仓库密码。
    2. 执行以下命令,创建Sercet。
      说明 Secret名字必需为asmwasm-cache,命名空间为istio-system。
      kubectl create secret generic asmwasm-cache -n istio-system --from-file=.dockerconfigjson=myconfig.json --type=kubernetes.io/dockerconfigjson
  2. 部署ASMFilterDeployment。
    1. 使用以下内容,创建名为hello1-afd.yaml的文件。
      apiVersion: istio.alibabacloud.com/v1beta1
      kind: ASMFilterDeployment
      metadata:
        name: hello1-propagate-header
      spec:
        workload:
          kind: Deployment
          labels:
            app: hello1-deploy-v2
            version: v2
        filter:
          patchContext: 'SIDECAR_OUTBOUND'
          parameters: '{"head_tag_name": "route-v", "head_tag_value": "hello2v2"}'
          image: 'wasm-repo-registry.cn-beijing.cr.aliyuncs.com/asm_wasm/propagate_header:0.0.1'
          rootID: 'propaganda_filter_root'
          id: 'hello1-propagate-header'
      • workload下的参数解释:
        1. kind:目标工作负载的类型。
        2. labels:筛选的条件。
      • filter下的参数解释:
        1. patchContext:生效的上下文阶段。
        2. parameters:运行WASM Filter所需的配置参数。
        3. image:WASM Fitler对应的镜像仓库地址。
        4. rootID:WASM Filter扩展插件对应的RootID。
        5. id:该WASM Filter的唯一ID。
    2. 使用以下内容,创建名为hello2-afd.yaml的文件。
      apiVersion: istio.alibabacloud.com/v1beta1
      kind: ASMFilterDeployment
      metadata:
        name: hello2-propagate-header
      spec:
        workload:
          kind: Deployment
          labels:
            app: hello2-deploy-v2
            version: v2
        filter:
          patchContext: 'SIDECAR_OUTBOUND'
          parameters: '{"head_tag_name": "route-v", "head_tag_value": "hello3v2"}'
          image: 'wasm-repo-registry.cn-beijing.cr.aliyuncs.com/asm_wasm/propagate_header:0.0.1'
          rootID: 'propaganda_filter_root'
          id: 'hello2-propagate-header'
      • workload下的参数解释:
        1. kind:目标工作负载的类型。
        2. labels:筛选的条件。
      • filter下的参数解释:
        1. patchContext:生效的上下文阶段。
        2. parameters:运行WASM Filter所需的配置参数。
        3. image:WASM Fitler对应的镜像仓库地址。
        4. rootID:WASM Filter扩展插件对应的RootID。
        5. id:该WASM Filter的唯一ID。
    3. 执行以下命令,部署ASMFilterDeployment。
      alias m="kubectl --kubeconfig $MESH_CONFIG"
      
      m apply -f hello1-afd.yaml -n "$NS"
      m apply -f hello2-afd.yaml -n "$NS"
  3. 执行以下命令,验证ASMFilterDeployment部署情况。
    ASMFilterDeployment部署后,会自动生成EnvoyFilter。
    alias m="kubectl --kubeconfig $MESH_CONFIG"
    
    m get envoyfilter -n "$NS"
    m get ASMFilterDeployment -n "$NS"

    预期输出:

    NAME                      AGE
    hello1-propagate-header   1s
    hello2-propagate-header   0s
    
    NAME                      STATUS      REASON   AGE
    hello1-propagate-header   Available            1s
    hello2-propagate-header   Available            1s

验证AB测试

执行以下命令,验证AB测试。

alias k="kubectl --kubeconfig $USER_CONFIG"

ingressGatewayIp=$(k -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
for j in {1..3}; do
  curl -H "route-v:v2" "http://$ingressGatewayIp:8001/hello/eric"
  echo
done

预期输出:

Bonjour eric@hello1:172.17.68.239<Bonjour eric@hello2:172.17.68.209<Bonjour eric@hello3:172.17.68.208
Bonjour eric@hello1:172.17.68.239<Bonjour eric@hello2:172.17.68.209<Bonjour eric@hello3:172.17.68.208
Bonjour eric@hello1:172.17.68.239<Bonjour eric@hello2:172.17.68.209<Bonjour eric@hello3:172.17.68.208

可以看到,当接收到route-v且版本为v2的请求时,该请求路由到hello1v2、hello2v2和hello3v2。

相关信息

如果您没有得到预期结果,您可以使用以下脚本检查服务的负载日志:
  • 检查Envoy日志
    alias k="kubectl --kubeconfig $USER_CONFIG"
    hello1_v2_pod=$(k get pod -l app=hello1-deploy-v2 -n "$NS" -o jsonpath={.items..metadata.name})
    # 修改envoy日志级别为info
    k -n "$NS" exec "$hello1_v2_pod" -c istio-proxy -- curl -XPOST -s "http://localhost:15000/logging?level=info"
    # 打印envoy日志
    k -n "$NS" logs -f deployment/hello1-deploy-v2 -c istio-proxy
  • 检查Hello服务日志
    lias k="kubectl --kubeconfig $USER_CONFIG"
    
    k -n "$NS" logs -f deployment/hello2-deploy-v1 -c hello-v1-deploy