Implement user-percentage-based canary releases with multi-tag routing using a hash tagging plugin-Container Compute Service(ACS)-阿里云帮助中心

Weight-based traffic splitting routes a percentage of *requests* — but it cannot guarantee that the same user always hits the same version. In a distributed microservice call chain, this breaks session consistency: a user who gets app-a v2 on one request may get app-a v1 on the next.

The hash tagging plug-in in Alibaba Cloud Service Mesh (ASM) solves this by tagging every request at the ingress gateway based on a stable user identifier (x-user-id), then propagating those tags through the entire call chain via ASMHeaderPropagation. Each downstream service reads the tag and routes to the correct version, giving you per-user session consistency across all services simultaneously.

This tutorial walks through a four-step canary release scenario:

Step	Action	Canary ratio
1	Deploy the baseline application (app-a v1, app-b v1, app-c v2)	—
2	Roll out app-a v2 and app-b v2	10% of users
3	Roll out app-c v3 (concurrent bug-fix release)	50% of users
4	Complete the app-c release and clean up	100% of users

Steps 2 and 3 run independent canary ratios on the same call chain simultaneously — which simple weight-based routing cannot express.

Background

A traffic lane defines which set of versioned services handles a given request end-to-end.

In the example above, two lanes exist across three services:

Version 1: app-a(V1), app-b(V1), app-c(V1)
Version 2: app-a(V2), app-b(V1), app-c(V2)

The ASM sidecar proxy routes requests within a lane by matching propagated headers. This works well when all services share the same canary ratio — but breaks down when independent teams need different ratios for different services at the same time.

Application architecture with version distribution

Consider this situation:

app-a and app-b are on v1. A new feature ships in v2, and the team wants to expose it to 10% of users.
app-c is on v2. A critical bug fix lands in v3, and the team wants to expose it to 50% of users as quickly as possible.

Simple weight-based routing cannot express these two independent ratios on a shared call chain. The hash tagging plug-in handles this by attaching multiple independent tags to each request at the gateway — one per service that needs canary routing. ASMHeaderPropagation then carries those tags through every hop in the call chain, and each VirtualService matches its own tag to pick the right version.

Prerequisites

Before you begin, make sure you have:

A cluster added to an ASM instance of version 1.18 or later. See Add a cluster to an ASM instance.
A Container Service for Kubernetes (ACK) managed cluster or an ACS cluster. See Create an ACK managed cluster or Create an ACS cluster.
An ingress gateway deployed. See Create an ingress gateway.
Automatic sidecar proxy injection enabled.

Step 1: Deploy the baseline application

The example uses three services in a linear call chain: app-a → app-b → app-c. The initial state has app-a and app-b on v1, and app-c on v2.

Create app-init.yaml with the following content.

app-init.yaml

apiVersion: v1
kind: Service
metadata:
  name: app-a
  labels:
    app: app-a
    service: app-a
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: app-a
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-a-v1
  labels:
    app: app-a
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-a
      version: v1
      ASM_TRAFFIC_TAG: v1
  template:
    metadata:
      labels:
        app: app-a
        version: v1
        ASM_TRAFFIC_TAG: v1
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v1
        - name: app
          value: app-a
        - name: upstream_url
          value: "http://app-b:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: app-b
  labels:
    app: app-b
    service: app-b
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: app-b
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-b-v1
  labels:
    app: app-b
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-b
      version: v1
      ASM_TRAFFIC_TAG: v1
  template:
    metadata:
      labels:
        app: app-b
        version: v1
        ASM_TRAFFIC_TAG: v1
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v1
        - name: app
          value: app-b
        - name: upstream_url
          value: "http://app-c:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: app-c
  labels:
    app: app-c
    service: app-c
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: app-c
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-c-v2
  labels:
    app: app-c
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-c
      version: v2
      ASM_TRAFFIC_TAG: v2
  template:
    metadata:
      labels:
        app: app-c
        version: v2
        ASM_TRAFFIC_TAG: v2
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v2
        - name: app
          value: app-c
        ports:
        - containerPort: 8000

Apply the manifest using the kubeconfig of the data plane cluster. This example uses the default namespace, but any namespace with automatic sidecar proxy injection enabled works.
```
kubectl apply -f app-init.yaml -n default
```

Create app-init-mesh.yaml with the following content.

app-init-mesh.yaml

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-b
  namespace: default
spec:
  hosts:
  - app-b.default.svc.cluster.local
  http:
  - name: default
    route:
    - destination:
        host: app-b.default.svc.cluster.local
        port:
          number: 8000
        subset: v1
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-c
  namespace: default
spec:
  hosts:
  - app-c.default.svc.cluster.local
  http:
  - name: default
    route:
    - destination:
        host: app-c.default.svc.cluster.local
        port:
          number: 8000
        subset: v2
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: ingressgateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*'
      port:
        name: http
        number: 80
        protocol: HTTP
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: ingressgateway
  namespace: istio-system
spec:
  gateways:
  - default/ingressgateway
  hosts:
  - '*'
  http:
  - name: default
    route:
    - destination:
        host: app-a.default.svc.cluster.local
        port:
          number: 8000
        subset: v1
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: app-a
  namespace: default
spec:
  host: app-a.default.svc.cluster.local
  subsets:
    - labels:
        version: v1
      name: v1
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: app-b
  namespace: default
spec:
  host: app-b.default.svc.cluster.local
  subsets:
    - labels:
        version: v1
      name: v1
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: app-c
  namespace: default
spec:
  host: app-c.default.svc.cluster.local
  subsets:
    - labels:
        version: v2
      name: v2

Apply the mesh configuration using the kubeconfig of the control plane cluster.
```
kubectl apply -f app-init-mesh.yaml
```
Verify that the baseline routing works. Replace ${ingress gateway ip} with the actual IP address of the ingress gateway. For instructions on getting the IP address, see Obtain the IP address of an ingress gateway.
```
curl -H 'x-user-id: 0001' ${ingress gateway ip}
```
Expected output:
```
-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v2, ip: 10.0.250.11)
```
The call chain app-a(v1) → app-b(v1) → app-c(v2) confirms baseline routing is working. Proceed to Step 2 to deploy the canary versions.

Step 2: Roll out app-a v2 and app-b v2 to 10% of users

This step deploys v2 for app-a and app-b, configures the hash tagging plug-in to tag 10% of users, and enables header propagation so the tags travel through the call chain.

The sub-steps below are ordered for clarity. In practice, adjust the order based on your service dependencies.

Create app-ab-v2.yaml with the following content.

app-ab-v2.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-a-v2
  labels:
    app: app-a
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-a
      version: v2
      ASM_TRAFFIC_TAG: v2
  template:
    metadata:
      labels:
        app: app-a
        version: v2
        ASM_TRAFFIC_TAG: v2
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v2
        - name: app
          value: app-a
        - name: upstream_url
          value: "http://app-b:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-b-v2
  labels:
    app: app-b
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-b
      version: v2
      ASM_TRAFFIC_TAG: v2
  template:
    metadata:
      labels:
        app: app-b
        version: v2
        ASM_TRAFFIC_TAG: v2
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v2
        - name: app
          value: app-b
        - name: upstream_url
          value: "http://app-c:8000/"
        ports:
        - containerPort: 8000

Deploy app-a v2 and app-b v2 using the kubeconfig of the data plane cluster.
```
kubectl apply -f app-ab-v2.yaml
```

Create app-ab-v2-mesh.yaml with the updated VirtualService and DestinationRule configurations. The VirtualServices add a header-match rule so requests tagged appver-a: v2 or appver-b: v2 route to the v2 subsets.

app-ab-v2-mesh.yaml

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-b
  namespace: default
spec:
  hosts:
  - app-b.default.svc.cluster.local
  http:
  - name: v2
    match:
    - headers:
        appver-b:
          exact: v2
    route:
    - destination:
        host: app-b.default.svc.cluster.local
        port:
          number: 8000
        subset: v2
  - name: default
    route:
    - destination:
        host: app-b.default.svc.cluster.local
        port:
          number: 8000
        subset: v1
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-c
  namespace: default
spec:
  hosts:
  - app-c.default.svc.cluster.local
  http:
  - name: default
    route:
    - destination:
        host: app-c.default.svc.cluster.local
        port:
          number: 8000
        subset: v2
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: ingressgateway
  namespace: default
spec:
  gateways:
  - default/ingressgateway
  hosts:
  - '*'
  http:
  - name: v2
    match:
    - headers:
        appver-a:
          exact: v2
    route:
    - destination:
        host: app-a.default.svc.cluster.local
        port:
          number: 8000
        subset: v2
  - name: default
    route:
    - destination:
        host: app-a.default.svc.cluster.local
        port:
          number: 8000
        subset: v1
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: app-a
spec:
  host: app-a.default.svc.cluster.local
  subsets:
    - labels:
        version: v1
      name: v1
    - labels:
        version: v2
      name: v2
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: app-b
spec:
  host: app-b.default.svc.cluster.local
  subsets:
    - labels:
        version: v1
      name: v1
    - labels:
        version: v2
      name: v2
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: app-c
spec:
  host: app-c.default.svc.cluster.local
  subsets:
    - labels:
        version: v2
      name: v2

Apply the mesh routing configuration using the kubeconfig of the control plane cluster.
```
kubectl apply -f app-ab-v2-mesh.yaml
```
Create header-propagation.yaml to configure ASMHeaderPropagation. This tells the sidecar proxy to propagate all request headers with the appver prefix through the call chain.
```
apiVersion: istio.alibabacloud.com/v1beta1
kind: ASMHeaderPropagation
metadata:
  name: tag-propagation
spec:
  headerPrefixes:
    - appver
```

Apply the header propagation configuration.

kubectl apply -f header-propagation.yaml -n default

Create hash-tagging-plugin.yaml to configure the WasmPlugin. The plug-in takes the x-user-id header value, computes hash(x-user-id) % 100, and adds tagging headers based on the result.

Rule 1: If hash(x-user-id) % 100 < 10, add appver-a: v2 to the request. This routes 10% of users to app-a v2.
Rule 2: If hash(x-user-id) % 100 < 10, add appver-b: v2 to the request. This routes the same 10% to app-b v2.

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: hash-tagging
  namespace: istio-system
spec:
  imagePullPolicy: IfNotPresent
  selector:
    matchLabels:
      istio: ingressgateway
  url: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-wasm-hash-tagging:v1.22.6.2-g72656ba-aliyun
  phase: AUTHN
  pluginConfig:
    rules:
      - header: x-user-id
        modulo: 100
        tagHeader: appver-a
        policies:
          - range: 10
            tagValue: v2
      - header: x-user-id
        modulo: 100
        tagHeader: appver-b
        policies:
          - range: 10
            tagValue: v2

The two rules work independently:

Apply the WasmPlugin configuration.

kubectl apply -f hash-tagging-plugin.yaml

Verify the canary routing by sending requests with different user IDs.

curl -H 'x-user-id: 0001' ${ingress gateway ip}
curl -H 'x-user-id: 0002' ${ingress gateway ip}
curl -H 'x-user-id: 0003' ${ingress gateway ip}
curl -H 'x-user-id: 0004' ${ingress gateway ip}
curl -H 'x-user-id: 0005' ${ingress gateway ip}

Expected output:

-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v2, ip: 10.0.250.11)
-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v2, ip: 10.0.250.11)
-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v2, ip: 10.0.250.11)
-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v2, ip: 10.0.250.11)
-> app-a(version: v2, ip: 10.0.250.14)-> app-b(version: v2, ip: 10.0.250.8)-> app-c(version: v2, ip: 10.0.250.11)

User 0005 falls within the 10% canary range. That user is tagged at the gateway and consistently routed to app-a v2 and app-b v2 — while app-c is unchanged at v2 for all users. The app-a and app-b canary is now running. When you confirm that v2 behavior is correct across your canary population, proceed to Step 3 — where you can independently roll out app-c v3 at a different ratio while this canary continues running.

Step 3: Roll out app-c v3 to 50% of users

While the app-a and app-b canary is running, this step adds an independent canary for app-c v3 — a bug-fix release for v2. Because the risk is low, the canary ratio is set to 50%. Each service's tagging rule is evaluated independently, so the app-c canary population is not tied to the app-a/app-b canary population.

Create app-c-v3.yaml with the following content.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-c-v3
  labels:
    app: app-c
    version: v3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-c
      version: v3
      ASM_TRAFFIC_TAG: v3
  template:
    metadata:
      labels:
        app: app-c
        version: v3
        ASM_TRAFFIC_TAG: v3
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v3
        - name: app
          value: app-c
        ports:
        - containerPort: 8000

Deploy app-c v3 using the kubeconfig of the data plane cluster.
```
kubectl apply -f app-c-v3.yaml
```

Create app-c-v3-mesh.yaml to add a v3 subset and a header-match routing rule for app-c.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-c
  namespace: default
spec:
  hosts:
  - app-c.default.svc.cluster.local
  http:
  - name: v3
    match:
    - headers:
        appver-c:
          exact: v3
    route:
    - destination:
        host: app-c.default.svc.cluster.local
        port:
          number: 8000
        subset: v3
  - name: default
    route:
    - destination:
        host: app-c.default.svc.cluster.local
        port:
          number: 8000
        subset: v2
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: app-c
spec:
  host: app-c.default.svc.cluster.local
  subsets:
    - labels:
        version: v2
      name: v2
    - labels:
        version: v3
      name: v3

Apply the mesh routing configuration using the kubeconfig of the control plane cluster.
```
kubectl apply -f app-c-v3-mesh.yaml
```

Create wasm-plugin-ab-v2-c-v3.yaml to update the WasmPlugin with a third tagging rule for app-c. The existing rules for app-a and app-b remain unchanged.

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: hash-tagging
  namespace: istio-system
spec:
  imagePullPolicy: IfNotPresent
  selector:
    matchLabels:
      istio: ingressgateway
  url: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-wasm-hash-tagging:v1.22.6.2-g72656ba-aliyun
  phase: AUTHN
  pluginConfig:
    rules:
      - header: x-user-id
        modulo: 100
        tagHeader: appver-a
        policies:
          - range: 10
            tagValue: v2
      - header: x-user-id
        modulo: 100
        tagHeader: appver-b
        policies:
          - range: 10
            tagValue: v2
      - header: x-user-id
        modulo: 100
        tagHeader: appver-c
        policies:
          - range: 50
            tagValue: v3

The third rule adds appver-c: v3 when hash(x-user-id) % 100 < 50, routing 50% of users to app-c v3. Each rule is evaluated independently: a user in the app-c v3 canary may or may not be in the app-a/app-b v2 canary.

Apply the updated WasmPlugin configuration.

kubectl apply -f wasm-plugin-ab-v2-c-v3.yaml

Verify all three canary ratios are working.

Users 0001 and 0002: app-a(v1) → app-b(v1) → app-c(v2) — outside both canary populations
Users 0003 and 0004: app-a(v1) → app-b(v1) → app-c(v3) — in the app-c v3 canary only
User 0005: app-a(v2) → app-b(v2) → app-c(v3) — in both canary populations

curl -H 'x-user-id: 0001' ${ingress gateway ip}
curl -H 'x-user-id: 0002' ${ingress gateway ip}
curl -H 'x-user-id: 0003' ${ingress gateway ip}
curl -H 'x-user-id: 0004' ${ingress gateway ip}
curl -H 'x-user-id: 0005' ${ingress gateway ip}

Expected output:

-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v2, ip: 10.0.250.11)
-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v2, ip: 10.0.250.11)
-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v3, ip: 10.0.250.23)
-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v3, ip: 10.0.250.23)
-> app-a(version: v2, ip: 10.0.250.14)-> app-b(version: v2, ip: 10.0.250.8)-> app-c(version: v3, ip: 10.0.250.23)

The three canary ranges are operating independently: When app-c v3 is stable, proceed to Step 4 to complete the release and clean up the tagging rules.

Step 4: Complete the app-c canary release

After validating app-c v3 in the canary environment, route all traffic to v3 and remove the now-unnecessary tagging rule.

Update the VirtualService for app-c to send all traffic to v3. Apply the following YAML using the kubeconfig of the control plane cluster.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-c
  namespace: default
spec:
  hosts:
  - app-c.default.svc.cluster.local
  http:
  - name: default
    route:
    - destination:
        host: app-c.default.svc.cluster.local
        port:
          number: 8000
        subset: v3

Remove the app-c tagging rule from the WasmPlugin to stop adding unnecessary headers to the call chain. Apply the following YAML using the kubeconfig of the ASM instance.

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: hash-tagging
  namespace: istio-system
spec:
  imagePullPolicy: IfNotPresent
  selector:
    matchLabels:
      istio: ingressgateway
  url: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-wasm-hash-tagging:v1.22.6.2-g72656ba-aliyun
  phase: AUTHN
  pluginConfig:
    rules:
      - header: x-user-id
        modulo: 100
        tagHeader: appver-a
        policies:
          - range: 10
            tagValue: v2
      - header: x-user-id
        modulo: 100
        tagHeader: appver-b
        policies:
          - range: 10
            tagValue: v2

Verify that all users are now routed to app-c v3.

After the cutover, set the replica count of app-c v2 to 0 or delete the deployment based on your requirements.

curl -H 'x-user-id: 0001' ${ingress gateway ip}
curl -H 'x-user-id: 0002' ${ingress gateway ip}
curl -H 'x-user-id: 0003' ${ingress gateway ip}
curl -H 'x-user-id: 0004' ${ingress gateway ip}
curl -H 'x-user-id: 0005' ${ingress gateway ip}

Expected output:

-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v3, ip: 10.0.250.23)
-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v3, ip: 10.0.250.23)
-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v3, ip: 10.0.250.23)
-> app-a(version: v1, ip: 10.0.250.27)-> app-b(version: v1, ip: 10.0.250.6)-> app-c(version: v3, ip: 10.0.250.23)
-> app-a(version: v2, ip: 10.0.250.14)-> app-b(version: v2, ip: 10.0.250.8)-> app-c(version: v3, ip: 10.0.250.23)

All users reach app-c v3. The canary release for app-c is complete.