Use EWMA for latency-based load balancing

Example overview

This example uses the simple-server application as the server and the sleep application as the client that generates test traffic. The simple-server.default.svc.cluster.local service, which is part of the simple-server application, acts as the server. This service includes two deployments with different configurations:

• simple-server-normal: This deployment is configured with a response latency of 50 ms to 100 ms.

• simple-server-high-latency: This deployment simulates a scenario where some workloads of a service experience increased latency.

Step 1: Enable ASM metrics

To demonstrate the effect of the peak EWMA load balancer, this example enables ASM metrics. This allows you to observe the change in overall service response time before and after the algorithm is enabled. For more information about how to enable metrics and collect them in Managed Service for Prometheus, see Collect metrics in Managed Service for Prometheus.

Step 2: Deploy the environment

1. In your KubeConfig environment for the ACK cluster, create a file named sleep.yaml with the following content.

   YAML content

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: sleep
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: sleep
     labels:
       app: sleep
       service: sleep
   spec:
     ports:
     - port: 80
       name: http
     selector:
       app: sleep
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: sleep
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: sleep
     template:
       metadata:
         labels:
           app: sleep
       spec:
         terminationGracePeriodSeconds: 0
         serviceAccountName: sleep
         containers:
         - name: sleep
           image: curlimages/curl
           command: ["/bin/sleep", "infinity"]
           imagePullPolicy: IfNotPresent
           volumeMounts:
           - mountPath: /etc/sleep/tls
             name: secret-volume
         volumes:
         - name: secret-volume
           secret:
             secretName: sleep-secret
             optional: true
   ---

   Run the following command to deploy the sleep application.

   kubectl apply -f sleep.yaml

2. Create a file named simple.yaml with the following content.

   YAML content

   apiVersion: v1
   kind: Service
   metadata:
     name: simple-server
     labels:
       app: simple-server
       service: simple-server
   spec:
     ports:
     - port: 8080
       name: http
     selector:
       app: simple-server
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: simple-server
     name: simple-server-normal
     namespace: default
   spec:
     progressDeadlineSeconds: 600
     replicas: 1
     revisionHistoryLimit: 10
     selector:
       matchLabels:
         app: simple-server
     strategy:
       rollingUpdate:
         maxSurge: 25%
         maxUnavailable: 25%
       type: RollingUpdate
     template:
       metadata:
         creationTimestamp: null
         labels:
           app: simple-server
       spec:
         containers:
         - args:
           - --delayMin
           - "50"
           - --delayMax
           - "100"
           image: registry-cn-hangzhou.ack.aliyuncs.com/test-public/simple-server:v1.0.0.0-g88293ca-aliyun
           imagePullPolicy: IfNotPresent
           name: simple-server
           ports:
           - containerPort: 80
             protocol: TCP
           resources:
             limits:
               cpu: 500m
           terminationMessagePath: /dev/termination-log
           terminationMessagePolicy: File
         dnsPolicy: ClusterFirst
         restartPolicy: Always
         schedulerName: default-scheduler
         securityContext: {}
         terminationGracePeriodSeconds: 30
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: simple-server
     name: simple-server-high-latency
     namespace: default
   spec:
     progressDeadlineSeconds: 600
     replicas: 1
     revisionHistoryLimit: 10
     selector:
       matchLabels:
         app: simple-server
     strategy:
       rollingUpdate:
         maxSurge: 25%
         maxUnavailable: 25%
       type: RollingUpdate
     template:
       metadata:
         creationTimestamp: null
         labels:
           app: simple-server
       spec:
         containers:
         - args:
           - --delayMin
           - "500"
           - --delayMax
           - "2000"
           image: registry-cn-hangzhou.ack.aliyuncs.com/test-public/simple-server:v1.0.0.0-g88293ca-aliyun
           imagePullPolicy: IfNotPresent
           name: simple-server
           ports:
           - containerPort: 80
             protocol: TCP
           resources:
             limits:
               cpu: 500m
           terminationMessagePath: /dev/termination-log
           terminationMessagePolicy: File
         dnsPolicy: ClusterFirst
         restartPolicy: Always
         schedulerName: default-scheduler
         securityContext: {}
         terminationGracePeriodSeconds: 30
   ---

   Run the following command to deploy the simple-server-normal and simple-server-high-latency applications.

   kubectl apply -f simple.yaml

Step 3: Run a baseline test

This section generates baseline data by using the default LEAST_REQUEST load balancing algorithm.

1. Run the following command to start the test. This command makes 100 calls to the /hello endpoint of the simple-server service:

   kubectl exec -it deploy/sleep -c sleep --  sh -c 'for i in $(seq 1 100); do time curl simple-server:8080/hello; echo "request $i done"; done'

   The expected output is as follows:

   hello
    this is port: 8080real 0m 0.06s
   user    0m 0.00s
   sys     0m 0.00s
   request 1 done
   hello
    this is port: 8080real 0m 0.09s
   user    0m 0.00s
   sys     0m 0.00s
   request 2 done
   
   ......
   
   hello
    this is port: 8080real 0m 1.72s
   user    0m 0.00s
   sys     0m 0.00s
   request 100 done

2. After the command finishes, on the Mesh Management page, click the name of your ASM instance. In the left-side navigation pane, choose Observability Management Center > Monitoring metrics. On the Cloud ASM Istio Service tab, enter the following filter conditions:

   • Namespace: default

   • Service: simple-server.default.svc.cluster.local

   • Reporter: destination

   • Client Workload Namespace: default

   • Client Workload: sleep

   • Service Workload Namespace: default

   • Service Workload: simple-server-normal + simple-server-high-latency

3. Click the Client Workloads drop-down menu and view the Incoming Request Duration By Source panel.

   

   The results show that the P50 response time for requests from the sleep application to the simple-server service is 87.5 ms, while the P95 response time increases significantly to 2.05 s. This is because the higher latency of simple-server-high-latency increases the overall response time of the service.

   Important

   The test results are theoretical values obtained in a controlled experimental environment. Actual results may vary depending on your specific workloads and environment.

Step 4: Configure peak EWMA and re-run the test

Create a DestinationRule to apply the peak EWMA load balancing algorithm to the simple-server service.

1. Follow the steps in Procedure and use the following YAML content to create the DestinationRule.

   apiVersion: networking.istio.io/v1beta1
   kind: DestinationRule
   metadata:
     name: simple-server
     namespace: default
   spec:
     host: simple-server.default.svc.cluster.local
     trafficPolicy:
       loadBalancer:
         simple: PEAK_EWMA

2. In your KubeConfig environment for the ACK cluster, run the test command again.

   kubectl exec -it deploy/sleep -c sleep --  sh -c 'for i in $(seq 1 100); do time curl simple-server:8080/hello; echo "request $i done"; done'

3. Comparing the results with the metrics from Step 3 shows that the P90, P95, and P99 response times decrease significantly. This is because the peak EWMA load balancer detects the higher latency of the simple-server-high-latency workload and reduces its load balancing weight. As a result, more requests are routed to the lower-latency simple-server-normal workload. This significantly reduces the overall request latency for the service.