本文介绍阿里云微服务引擎MSE在应用发布时提供的无损上下线和服务预热能力。假设应用的架构由Zuul网关以及后端的微服务应用实例(Spring Cloud)构成,具体的后端调用链路有购物车应用A,交易中心应用B,库存中心应用C,这些应用中的服务之间通过Nacos注册中心实现服务注册与发现。

前提条件

背景信息

很多请求量大并发度高的应用系统为了避免发布过程中流量有损,一般选择在流量较小的时候发布。虽然可以达到一定的效果,但不可控因素所产生的运维成本也很高。基于此,阿里云微服务引擎MSE在应用发布过程中,提供了应用下线时自适应等待和主动通知、应用上线时就绪检查与微服务生命周期对齐以及服务预热等技术手段,实现微服务应用无损上下线发布,有效规避线上发布所出现的流量损失,降低研发运维成本。

准备工作

流量压力来源

在spring-cloud-zuul应用中,如下图所示,其分别向spring-cloud-a的灰度版本和正常版本以QPS为100的速率同时进行服务调用。应用部署流量架构图

部署Demo应用程序

本示例Demo文件取名为mse-demo.yaml

注意 由于Demo中有CronHPA任务,所以请先在集群中安装ack-kubernetes-cronhpa-controller组件。具体操作,请参见安装CronHPA组件
  1. 容器服务控制台左侧导航栏中,单击集群
  2. 集群列表页面中,单击目标集群名称或者目标集群右侧操作列下的详情
  3. 在集群管理页左侧导航栏中,选择工作负载 > 无状态
  4. 无状态页面选择命名空间,然后单击使用YAML创建资源
  5. 对模板进行相关配置,完成配置后单击创建
    本文示例中部署Zuul网关和A、B、C三个应用,其中A、B两个应用分别部署一个基线版本和一个灰度版本,B应用的基线版本关闭了无损下线能力,灰度版本开启了无损下线能力,C应用开启了服务预热能力,其中预热时长为120秒。
    # Nacos Server
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: nacos-server
      name: nacos-server
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: nacos-server
      template:
        metadata:
          labels:
            app: nacos-server
            msePilotCreateAppName: nacos-server
            msePilotAutoEnable: "on"
        spec:
          containers:
          - env:
            - name: MODE
              value: standalone
            image: registry.cn-shanghai.aliyuncs.com/yizhan/nacos-server:latest
            imagePullPolicy: Always
            name: nacos-server
            resources:
              requests:
                cpu: 250m
                memory: 512Mi
          dnsPolicy: ClusterFirst
          restartPolicy: Always
    
    # Nacos Server Service配置
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: nacos-server
    spec:
      ports:
      - port: 8848
        protocol: TCP
        targetPort: 8848
      selector:
        app: nacos-server
      type: ClusterIP
    
    #入口Zuul应用
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: spring-cloud-zuul
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: spring-cloud-zuul
      template:
        metadata:
          labels:
            app: spring-cloud-zuul
            msePilotCreateAppName: spring-cloud-zuul
            msePilotAutoEnable: "on"
        spec:
          containers:
            - env:
                - name: JAVA_HOME
                  value: /usr/lib/jvm/java-1.8-openjdk/jre
                - name: LANG
                  value: C.UTF-8
              image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-zuul:1.0.1
              imagePullPolicy: Always
              name: spring-cloud-zuul
              ports:
                - containerPort: 20000
    
    # A应用base版本,开启按照机器纬度全链路透传。
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: spring-cloud-a
      name: spring-cloud-a
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: spring-cloud-a
      template:
        metadata: 
          labels:
            app: spring-cloud-a
            msePilotCreateAppName: spring-cloud-a
            msePilotAutoEnable: "on"
        spec:
          containers:
          - env:
            - name: LANG
              value: C.UTF-8
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
            - name: profiler.micro.service.tag.trace.enable
              value: "true"
            image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:0.1-SNAPSHOT
            imagePullPolicy: Always
            name: spring-cloud-a
            ports:
            - containerPort: 20001
              protocol: TCP
            resources:
              requests:
                cpu: 250m
                memory: 512Mi
            livenessProbe:
              tcpSocket:
                port: 20001
              initialDelaySeconds: 10
              periodSeconds: 30
    
    # A应用gray版本,开启按照机器纬度全链路透传。
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: spring-cloud-a-gray
      name: spring-cloud-a-gray
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: spring-cloud-a-gray
      strategy:
      template:
        metadata:
          annotations:
            alicloud.service.tag: gray
          labels:
            app: spring-cloud-a-gray
            msePilotCreateAppName: spring-cloud-a
            msePilotAutoEnable: "on"
        spec:
          containers:
          - env:
            - name: LANG
              value: C.UTF-8
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
            - name: profiler.micro.service.tag.trace.enable
              value: "true"
            image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:0.1-SNAPSHOT
            imagePullPolicy: Always
            name: spring-cloud-a-gray
            ports:
            - containerPort: 20001
              protocol: TCP
            resources:
              requests:
                cpu: 250m
                memory: 512Mi
            livenessProbe:
              tcpSocket:
                port: 20001
              initialDelaySeconds: 10
              periodSeconds: 30
    
    # B应用base版本,关闭无损下线能力。
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: spring-cloud-b
      name: spring-cloud-b
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: spring-cloud-b
      strategy:
      template:
        metadata:
          labels:
            app: spring-cloud-b
            msePilotCreateAppName: spring-cloud-b
            msePilotAutoEnable: "on"
        spec:
          containers:
          - env:
            - name: LANG
              value: C.UTF-8
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
            - name: micro.service.shutdown.server.enable
              value: "false"
            - name: profiler.micro.service.http.server.enable
              value: "false"
            image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:0.1-SNAPSHOT
            imagePullPolicy: Always
            name: spring-cloud-b
            ports:
            - containerPort: 8080
              protocol: TCP
            resources:
              requests:
                cpu: 250m
                memory: 512Mi
            livenessProbe:
              tcpSocket:
                port: 20002
              initialDelaySeconds: 10
              periodSeconds: 30
    
    # B应用gray版本,默认开启无损下线功能。
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: spring-cloud-b-gray
      name: spring-cloud-b-gray
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: spring-cloud-b-gray
      template:
        metadata:
          annotations:
            alicloud.service.tag: gray
          labels:
            app: spring-cloud-b-gray
            msePilotCreateAppName: spring-cloud-b
            msePilotAutoEnable: "on"
        spec:
          containers:
          - env:
            - name: LANG
              value: C.UTF-8
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
            image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:0.1-SNAPSHOT
            imagePullPolicy: Always
            name: spring-cloud-b-gray
            ports:
            - containerPort: 8080
              protocol: TCP
            resources:
              requests:
                cpu: 250m
                memory: 512Mi
            lifecycle:
                preStop:
                  exec:
                    command:
                      - /bin/sh
                      - '-c'
                      - >-
                        wget http://127.0.0.1:54199/offline 2>/tmp/null;sleep
                        30;exit 0
            livenessProbe:
              tcpSocket:
                port: 20002
              initialDelaySeconds: 10
              periodSeconds: 30
    
    # C应用base版本
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: spring-cloud-c
      name: spring-cloud-c
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: spring-cloud-c
      template:
        metadata:
          labels:
            app: spring-cloud-c
            msePilotCreateAppName: spring-cloud-c
            msePilotAutoEnable: "on"
        spec:
          containers:
          - env:
            - name: LANG
              value: C.UTF-8
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
            image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:0.1-SNAPSHOT
            imagePullPolicy: Always
            name: spring-cloud-c
            ports:
            - containerPort: 8080
              protocol: TCP
            resources:
              requests:
                cpu: 250m
                memory: 512Mi
            livenessProbe:
              tcpSocket:
                port: 20003
              initialDelaySeconds: 10
              periodSeconds: 30
    
    # HPA配置
    ---
    apiVersion: autoscaling.alibabacloud.com/v1beta1
    kind: CronHorizontalPodAutoscaler
    metadata:
      labels:
        controller-tools.k8s.io: "1.0"
      name: spring-cloud-b
    spec:
       scaleTargetRef:
          apiVersion: apps/v1beta2
          kind: Deployment
          name: spring-cloud-b
       jobs:
       - name: "scale-down"
         schedule: "0 0/5 * * * *"
         targetSize: 1
       - name: "scale-up"
         schedule: "10 0/5 * * * *"
         targetSize: 2
    ---
    apiVersion: autoscaling.alibabacloud.com/v1beta1
    kind: CronHorizontalPodAutoscaler
    metadata:
      labels:
        controller-tools.k8s.io: "1.0"
      name: spring-cloud-b-gray
    spec:
       scaleTargetRef:
          apiVersion: apps/v1beta2
          kind: Deployment
          name: spring-cloud-b-gray
       jobs:
       - name: "scale-down"
         schedule: "0 0/5 * * * *"
         targetSize: 1
       - name: "scale-up"
         schedule: "10 0/5 * * * *"
         targetSize: 2
    ---
    apiVersion: autoscaling.alibabacloud.com/v1beta1
    kind: CronHorizontalPodAutoscaler
    metadata:
      labels:
        controller-tools.k8s.io: "1.0"
      name: spring-cloud-c
    spec:
       scaleTargetRef:
          apiVersion: apps/v1beta2
          kind: Deployment
          name: spring-cloud-c
       jobs:
       - name: "scale-down"
         schedule: "0 2/5 * * * *"
         targetSize: 1
       - name: "scale-up"
         schedule: "10 2/5 * * * *"
         targetSize: 2
    
    
    # Zuul网关开启SLB暴露展示页面
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: zuul-slb
    spec:
      ports:
        - port: 80
          protocol: TCP
          targetPort: 20000
      selector:
        app: spring-cloud-zuul
      type: ClusterIP
    
    # A应用暴露K8s Service
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: spring-cloud-a-base
    spec:
      ports:
        - name: http
          port: 20001
          protocol: TCP
          targetPort: 20001
      selector:
        app: spring-cloud-a
    
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: spring-cloud-a-gray
    spec:
      ports:
        - name: http
          port: 20001
          protocol: TCP
          targetPort: 20001
      selector:
        app: spring-cloud-a-gray
    
    # Nacos Server SLB Service配置
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: nacos-slb
    spec:
      ports:
      - port: 8848
        protocol: TCP
        targetPort: 8848
      selector:
        app: nacos-server
      type: LoadBalancer
  6. 执行以下命令:
    kubectl apply -f mse-demo.yaml
    预期输出:
    deployment.apps/nacos-server created
    service/nacos-server created
    deployment.apps/spring-cloud-zuul created
    deployment.apps/spring-cloud-a created
    deployment.apps/spring-cloud-a-gray created
    deployment.apps/spring-cloud-b created
    deployment.apps/spring-cloud-b-gray created
    deployment.apps/spring-cloud-c created
    cronhorizontalpodautoscaler.autoscaling.alibabacloud.com/spring-cloud-b created
    cronhorizontalpodautoscaler.autoscaling.alibabacloud.com/spring-cloud-b-gray created
    cronhorizontalpodautoscaler.autoscaling.alibabacloud.com/spring-cloud-c created
    service/zuul-slb created
    service/spring-cloud-a-base created
    service/spring-cloud-a-gray created
    service/nacos-slb created

结果验证:无损下线功能

由于spring-cloud-b应用和spring-cloud-b-gray应用均开启了定时HPA,模拟每5分钟进行一次定时的扩缩容。

  • spring-cloud-b应用:spring-cloud-b应用
  • spring-cloud-b-gray应用:spring-cloud-b-gray
  1. 登录MSE治理中心控制台
  2. 在顶部菜单栏选择地域。
  3. 在左侧导航栏选择微服务治理中心 > 无损上下线
  4. 应用列表列单击spring-cloud-a应用名称,右侧显示该应用相关的可观测数据。
    spring-cloud-a
    从数据中可以看出:
    • spring-cloud-a-gray版本的流量在Pod扩缩容的过程中请求错误数为0,无流量损失。
    • spring-cloud-a版本由于关闭了无损下线功能,在Pod扩缩容的过程中有20个从spring-cloud-a发到spring-cloud-b的请求出现报错,发生了请求流量损耗。

结果验证:服务预热功能

在spring-cloud-c应用开启了定时HPA模拟应用启动的过程,每隔5分钟做一次伸缩,在第2分钟第0秒缩容到1个节点,在第2分钟第10秒扩容到2个节点。spring-cloud-c
  1. 登录MSE治理中心控制台
  2. 在顶部菜单栏选择地域。
  3. 在左侧导航栏选择微服务治理中心 > 无损上下线
  4. 应用列表列单击spring-cloud-c应用名称,在应用无损上线配置区域打开预热开关,预热时长设置为120秒。具体操作,请参见小流量预热服务
    服务预热
    观察节点的流量,发现节点流量缓慢上升,并且能看到节点的预热开始和结束时间,以及相关的事件。预热结果

    从数据可以看出,开启预热功能的应用重启后的流量会随时间缓慢增加,在一些应用启动过程中需要预建连接池和缓存等资源的慢启动场景,开启服务预热能有效保护应用启动过程中缓存资源有序创建保障应用安全启动并做到流量无损。