使用ASMAdaptiveConcurrency实现自适应并发控制

ASMAdaptiveConcurrency能够根据采样的请求数据动态调整允许的并发的数量,当并发数量超过服务可承受范围时,会拒绝请求,达到保护服务的目的。本文介绍如何使用ASMAdaptiveConcurrency实现自适应并发控制。

前提条件

背景信息

通常情况下,希望服务在超出负载能力时能拒绝超额的请求,从而防止产生其他连锁反应。您可以使用服务网格的DestinationRule配置基础的熔断能力,但这要求必须给出一个触发熔断的阈值,例如具体的pending requests数量,使得服务网格数据平面在网络访问超出熔断配置时能够拒绝请求。在实际应用场景下,准确地估算服务的负载能力是较为困难的。

ASMAdaptiveConcurrency采用自适应并发控制算法通过定期对比采样Latency和测定的理想Latency之间的差异及一系列计算对并发限制进行动态调整,从而尽可能使得并发限制数量在服务可承受的范围附近,同时拒绝超出该限制的请求(拒绝请求时会返回HTTP 503及错误信息reached concurrency limit)。

由于周期性的MinRTT统计期间会将连接数限制在较小值(min_concurrency),建议您在启用了AdaptiveConcurrency服务后,同时使用DestinationRule为服务启用重试功能,使得在minRTT计算期间被拒绝的请求能够通过Sidecar的重试功能尽可能地成功返回结果。

步骤一:部署示例应用

部署testserver和gotest应用,设置testserver应用的负载能力为并发处理500个请求,超过并发数量的请求会被排队处理。每个请求的处理时间是1000ms。设置gotest应用的一个副本一次能发起200个请求。

  1. 部署testserver应用。

    1. 使用以下内容,创建testserver.yaml文件。

      展开查看详细内容

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        labels:
          app: testserver
        name: testserver
        namespace: default
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: testserver
        template:
          metadata:
            creationTimestamp: null
            labels:
              app: testserver
          spec:
            containers:
            - args:
              - -m
              - "500"
              - -t
              - "1000"
              command:
              - /usr/local/bin/limited-concurrency-http-server
              image: registry.cn-hangzhou.aliyuncs.com/acs/asm-limited-concurrency-http-server:v0.1.1-gee0b08f-aliyun
              imagePullPolicy: IfNotPresent
              name: testserver
              ports:
              - containerPort: 8080
                protocol: TCP

      您可以使用-m指定应用能够承受的并发数量,-t指定请求处理时间。

    2. 执行以下命令,部署testserver应用。

      kubectl apply -f testserver.yaml
  2. 部署testserver的Service。

    1. 使用以下内容,创建testservice.yaml文件。

      展开查看详细内容

      apiVersion: v1
      kind: Service
      metadata:
        labels:
          app: testserver
        name: testserver
        namespace: default
      spec:
        internalTrafficPolicy: Cluster
        ipFamilies:
          - IPv4
        ipFamilyPolicy: SingleStack
        ports:
          - name: http
            port: 8080
            protocol: TCP
            targetPort: 8080
          - name: metrics
            port: 15020
            protocol: TCP
            targetPort: 15020
        selector:
          app: testserver
        type: ClusterIP
                                      
    2. 执行以下命令,部署testserver的Service。

      kubectl apply -f testservice.yaml
  3. 部署gotest应用。

    1. 使用以下内容,创建gotest.yaml文件。

      展开查看详细内容

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        labels:
          app: gotest
        name: gotest
        namespace: default
      spec:
        replicas: 0
        selector:
          matchLabels:
            app: gotest
        template:
          metadata:
            creationTimestamp: null
            labels:
              app: gotest
          spec:
            containers:
            - args:
              - -c
              - "200"
              - -n
              - "10000"
              - -u
              - testserver:8080
              command:
              - /root/go-stress-testing-linux
              image: xocoder/go-stress-testing-linux:v0.1
              imagePullPolicy: Always
              name: gotest
              resources:
                limits:
                  cpu: 500m
    2. 执行以下命令,部署gotest应用。

      kubectl apply -f gotest.yaml

步骤二:创建ASMAdaptiveConcurrency

  1. 通过控制面kubectl访问Istio资源

  2. 使用以下内容,创建adaptiveconcurrency.yaml

    展开查看详细内容

    apiVersion: istio.alibabacloud.com/v1beta1
    kind: ASMAdaptiveConcurrency
    metadata:
        name: sample-adaptive-concurrency
        namespace: default
    spec:
        workload_selector:
            labels:
                app: testserver
        sample_aggregate_percentile:
            value: 60
        concurrency_limit_params:
            max_concurrency_limit: 500
            concurrency_update_interval: 15s
        min_rtt_calc_params:
            interval: 60s
            request_count: 100
            jitter:
                value: 15
            min_concurrency: 50
            buffer:
                value: 25

    参数

    类型

    描述

    是否必填

    workload_selector

    WorkloadSelector

    工作负载选择器,声明如何匹配生效的工作负载。

    labels

    map

    列出需要匹配的工作负载(Pod)的Labels,用于匹配工作负载。

    sample_aggregate_percentile

    Percent

    采样百分位,将采样请求的该百分位值作为SampleRTT参与计算。

    value

    int

    百分比数,有效值为0~100。

    concurrency_limit_params

    ConcurrencyLimitParams

    并发限制相关配置。

    max_concurrency_limit

    int

    最大并发限制数。默认值1000。

    concurrency_update_interval

    duration

    最大并发计算间隔,例如60s。

    min_rtt_calc_params

    MinRTTCalcParams

    计算MinRTT的相关配置。

    interval

    duration

    理想round-trip time计算间隔,例如120s。

    request_count

    int

    统计多少请求的数据用作计算理想round-trip time。默认值为50。

    jitter

    Percent

    计算理想round-trip time时,增加随机延迟的百分比,例如,interval为120s时,jitter为50,则间隔时间为ramdom(120, 120 + (120 * 50%))。默认值为15。

    min_concurrency

    int

    计算理想round-trip time时的并发数量,同时也是控制器初始的并发数值。该值的取值应当远低于服务的瓶颈,从而使得服务在该并发下可以以最理想的延迟返回。默认值为3。

    buffer

    Percent

    合理延迟波动范围(百分比),例如延迟在100ms上下时,10%浮动属于合理范围,则这个值设置为10。默认值为25。

  3. 执行以下命令,创建ASMAdaptiveConcurrency。

    kubectl apply -f adaptiveconcurrency.yaml

步骤三:启用可观测监控Prometheus版

为了对并发控制器的运行状态有直观的理解,以及便于对参数调优,您可以将自适应并发控制的相关Metrics输出到可观测监控Prometheus版,使用可观测监控Prometheus版查看并发控制器的运行状态。

  1. 启用可观测监控Prometheus版。具体操作,请参见使用阿里云Prometheus监控

  2. 在集群中配置ServiceMonitor,使可观测监控Prometheus版可以获取到testserver数据。

    1. 使用以下内容,创建servicemonitor.yaml。

      apiVersion: monitoring.coreos.com/v1
      kind: ServiceMonitor
      metadata:
        name: testserver-envoy-metrics
        namespace: default
      spec:
        endpoints:
          - interval: 5s
            path: /stats/prometheus
            port: metrics
        namespaceSelector:
          any: true
        selector:
          matchLabels:
            app: testserver
    2. 在集群对应的KubeConfig环境下,执行以下命令,创建ServiceMonitor。

      kubectl apply -f servicemonitor.yaml

步骤四:验证ASMAdaptiveConcurrency的自适应并发控制是否成功

  1. 设置gotest的副本数为5。

    一个gotest应用副本将发起200个请求,5个副本则将发起1000个请求。

    1. 登录容器服务管理控制台

    2. 在控制台左侧导航栏,单击集群

    3. 集群列表页面,单击目标集群名称或者目标集群右侧操作列下的详情

    4. 在集群管理页左侧导航栏,选择工作负载 > 无状态

    5. 无状态页面设置命名空间为default,在gotest应用右侧操作列下选择更多 > 查看Yaml

    6. 编辑 YAML对话框中,设置replicas5,单击更新

  2. 使用如下JSON定义,为Grafana导入Dashboard,查看并发控制器的运行状态。具体操作,请参见导入仪表盘

    展开查看JSON定义

    {
      "annotations": {
        "list": [
          {
            "builtIn": 1,
            "datasource": "-- Grafana --",
            "enable": true,
            "hide": true,
            "iconColor": "rgba(0, 211, 255, 1)",
            "name": "Annotations & Alerts",
            "type": "dashboard"
          }
        ]
      },
      "description": "monitoring ASM Adaptive Concurrency",
      "editable": true,
      "gnetId": 6693,
      "graphTooltip": 0,
      "id": 3239002,
      "iteration": 1651922323976,
      "links": [],
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "$cluster",
          "fieldConfig": {
            "defaults": {
              "custom": {}
            },
            "overrides": []
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 0
          },
          "hiddenSeries": false,
          "id": 22,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "7.4.0-pre",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_rq_blocked{service=\"$service\", pod=\"$pod\"}",
              "interval": "",
              "legendFormat": "{{service}}-{{pod}}",
              "queryType": "randomWalk",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeRegions": [],
          "timeShift": null,
          "title": "RqBlocked",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "$cluster",
          "fieldConfig": {
            "defaults": {
              "custom": {}
            },
            "overrides": []
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 0
          },
          "hiddenSeries": false,
          "id": 24,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "7.4.0-pre",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_burst_queue_size{service=\"$service\", pod=\"$pod\"}",
              "format": "time_series",
              "interval": "",
              "legendFormat": "{{service}}-{{pod}}",
              "queryType": "randomWalk",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeRegions": [],
          "timeShift": null,
          "title": "HeadRoom",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "$cluster",
          "fieldConfig": {
            "defaults": {
              "custom": {}
            },
            "overrides": []
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 8
          },
          "hiddenSeries": false,
          "id": 26,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "7.4.0-pre",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_concurrency_limit{service=\"$service\",pod=\"$pod\"}",
              "interval": "",
              "legendFormat": "{{service}}-{{pod}}",
              "queryType": "randomWalk",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeRegions": [],
          "timeShift": null,
          "title": "ConcurrencyLimit",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "$cluster",
          "fieldConfig": {
            "defaults": {
              "custom": {}
            },
            "overrides": []
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 8
          },
          "hiddenSeries": false,
          "id": 28,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "7.4.0-pre",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_gradient{service=\"$service\",pod=\"$pod\"}",
              "interval": "",
              "legendFormat": "{{service}}-{{pod}}",
              "queryType": "randomWalk",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeRegions": [],
          "timeShift": null,
          "title": "Gradient",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "$cluster",
          "fieldConfig": {
            "defaults": {
              "custom": {}
            },
            "overrides": []
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 16
          },
          "hiddenSeries": false,
          "id": 32,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "7.4.0-pre",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_min_rtt_msecs{service=\"$service\",pod=\"$pod\"}",
              "interval": "",
              "legendFormat": "{{service}}-{{pod}}",
              "queryType": "randomWalk",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeRegions": [],
          "timeShift": null,
          "title": "MinRTT(msec)",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ms",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "$cluster",
          "fieldConfig": {
            "defaults": {
              "custom": {}
            },
            "overrides": []
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 12,
            "y": 16
          },
          "hiddenSeries": false,
          "id": 34,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "7.4.0-pre",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_sample_rtt_msecs{service=\"$service\",pod=\"$pod\"}",
              "interval": "",
              "legendFormat": "{{service}}-{{pod}}",
              "queryType": "randomWalk",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeRegions": [],
          "timeShift": null,
          "title": "SampleRTT(msec)",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ms",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "test-adaptive-concurrency_1217520382582089",
          "fieldConfig": {
            "defaults": {
              "custom": {}
            },
            "overrides": []
          },
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 24
          },
          "hiddenSeries": false,
          "id": 30,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "alertThreshold": true
          },
          "percentage": false,
          "pluginVersion": "7.4.0-pre",
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_min_rtt_calculation_active{service=\"$service\",pod=\"$pod\"}",
              "interval": "",
              "legendFormat": "{{service}}-{{pod}}",
              "queryType": "randomWalk",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeRegions": [],
          "timeShift": null,
          "title": "MinRTTCalc",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "refresh": "5s",
      "schemaVersion": 26,
      "style": "dark",
      "tags": [],
      "templating": {
        "list": [
          {
            "current": {
              "selected": true,
              "text": "edas120_1217520382582089",
              "value": "edas120_1217520382582089"
            },
            "error": null,
            "hide": 0,
            "includeAll": false,
            "label": null,
            "multi": false,
            "name": "cluster",
            "options": [],
            "query": "prometheus",
            "queryValue": "",
            "refresh": 1,
            "regex": "",
            "skipUrlSync": false,
            "type": "datasource"
          },
          {
            "allValue": null,
            "current": {
              "isNone": true,
              "selected": false,
              "text": "None",
              "value": ""
            },
            "datasource": "$cluster",
            "definition": "label_values(envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_burst_queue_size,service)",
            "error": null,
            "hide": 0,
            "includeAll": false,
            "label": null,
            "multi": false,
            "name": "service",
            "options": [],
            "query": "label_values(envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_burst_queue_size,service)",
            "refresh": 2,
            "regex": "",
            "skipUrlSync": false,
            "sort": 1,
            "tagValuesQuery": "",
            "tags": [],
            "tagsQuery": "",
            "type": "query",
            "useTags": false
          },
          {
            "allValue": null,
            "current": {
              "selected": false,
              "text": "All",
              "value": "$__all"
            },
            "datasource": "$cluster",
            "definition": "label_values(envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_concurrency_limit, pod)",
            "error": null,
            "hide": 0,
            "includeAll": true,
            "label": null,
            "multi": true,
            "name": "pod",
            "options": [],
            "query": "label_values(envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_concurrency_limit, pod)",
            "refresh": 2,
            "regex": "",
            "skipUrlSync": false,
            "sort": 0,
            "tagValuesQuery": "",
            "tags": [],
            "tagsQuery": "",
            "type": "query",
            "useTags": false
          }
        ]
      },
      "time": {
        "from": "now-15m",
        "to": "now"
      },
      "timepicker": {
        "refresh_intervals": [
          "5s",
          "10s",
          "30s",
          "1m",
          "5m",
          "15m",
          "30m",
          "1h",
          "2h",
          "1d"
        ],
        "time_options": [
          "5m",
          "15m",
          "1h",
          "6h",
          "12h",
          "24h",
          "2d",
          "7d",
          "30d"
        ]
      },
      "timezone": "",
      "title": "ASM Adaptive Concurrency",
      "uid": "000000084",
      "version": 3
    }
  3. 在Dashboard中选择集群,设置service为testserver,pod为ALL。

    gotest应用向testserver应用发起了1000个请求,但是testserver应用始终限制收到的并发数在500以内,表明使用ASMAdaptiveConcurrency的自适应并发控制成功。