本文介绍阿里云微服务引擎MSE在应用发布时提供的无损上下线和服务预热能力。假设应用的架构由Zuul网关以及后端的微服务应用实例(Spring Cloud)构成,具体的后端调用链路有购物车应用A,交易中心应用B,库存中心应用C,这些应用中的服务之间通过Nacos注册中心实现服务注册与发现。
背景信息
很多请求量大并发度高的应用系统为了避免发布过程中流量有损,一般选择在流量较小的时候发布。虽然可以达到一定的效果,但不可控因素所产生的运维成本也很高。基于此,阿里云微服务引擎MSE在应用发布过程中,提供了应用下线时自适应等待和主动通知、应用上线时就绪检查与微服务生命周期对齐以及服务预热等技术手段,实现微服务应用无损上下线发布,有效规避线上发布所出现的流量损失,降低研发运维成本。
准备工作
流量压力来源
在spring-cloud-zuul应用中,如下图所示,其分别向spring-cloud-a的灰度版本和正常版本以QPS为100的速率同时进行服务调用。

部署Demo应用程序
本示例Demo文件取名为mse-demo.yaml。
注意 由于Demo中有CronHPA任务,所以请先在集群中安装ack-kubernetes-cronhpa-controller组件。具体操作,请参见
安装CronHPA组件。
- 在容器服务控制台左侧导航栏中,单击集群。
- 在集群列表页面中,单击目标集群名称或者目标集群右侧操作列下的详情。
- 在集群管理页左侧导航栏中,选择。
- 在无状态页面选择命名空间,然后单击使用YAML创建资源。
- 对模板进行相关配置,完成配置后单击创建。
本文示例中部署Zuul网关和A、B、C三个应用,其中A、B两个应用分别部署一个基线版本和一个灰度版本,B应用的基线版本关闭了无损下线能力,灰度版本开启了无损下线能力,C应用开启了服务预热能力,其中预热时长为120秒。
# Nacos Server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nacos-server
name: nacos-server
spec:
replicas: 1
selector:
matchLabels:
app: nacos-server
template:
metadata:
labels:
app: nacos-server
msePilotCreateAppName: nacos-server
msePilotAutoEnable: "on"
spec:
containers:
- env:
- name: MODE
value: standalone
image: registry.cn-shanghai.aliyuncs.com/yizhan/nacos-server:latest
imagePullPolicy: Always
name: nacos-server
resources:
requests:
cpu: 250m
memory: 512Mi
dnsPolicy: ClusterFirst
restartPolicy: Always
# Nacos Server Service配置
---
apiVersion: v1
kind: Service
metadata:
name: nacos-server
spec:
ports:
- port: 8848
protocol: TCP
targetPort: 8848
selector:
app: nacos-server
type: ClusterIP
#入口Zuul应用
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: spring-cloud-zuul
spec:
replicas: 1
selector:
matchLabels:
app: spring-cloud-zuul
template:
metadata:
labels:
app: spring-cloud-zuul
msePilotCreateAppName: spring-cloud-zuul
msePilotAutoEnable: "on"
spec:
containers:
- env:
- name: JAVA_HOME
value: /usr/lib/jvm/java-1.8-openjdk/jre
- name: LANG
value: C.UTF-8
image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-zuul:1.0.1
imagePullPolicy: Always
name: spring-cloud-zuul
ports:
- containerPort: 20000
# A应用base版本,开启按照机器纬度全链路透传。
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: spring-cloud-a
name: spring-cloud-a
spec:
replicas: 2
selector:
matchLabels:
app: spring-cloud-a
template:
metadata:
labels:
app: spring-cloud-a
msePilotCreateAppName: spring-cloud-a
msePilotAutoEnable: "on"
spec:
containers:
- env:
- name: LANG
value: C.UTF-8
- name: JAVA_HOME
value: /usr/lib/jvm/java-1.8-openjdk/jre
- name: profiler.micro.service.tag.trace.enable
value: "true"
image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:0.1-SNAPSHOT
imagePullPolicy: Always
name: spring-cloud-a
ports:
- containerPort: 20001
protocol: TCP
resources:
requests:
cpu: 250m
memory: 512Mi
livenessProbe:
tcpSocket:
port: 20001
initialDelaySeconds: 10
periodSeconds: 30
# A应用gray版本,开启按照机器纬度全链路透传。
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: spring-cloud-a-gray
name: spring-cloud-a-gray
spec:
replicas: 2
selector:
matchLabels:
app: spring-cloud-a-gray
strategy:
template:
metadata:
annotations:
alicloud.service.tag: gray
labels:
app: spring-cloud-a-gray
msePilotCreateAppName: spring-cloud-a
msePilotAutoEnable: "on"
spec:
containers:
- env:
- name: LANG
value: C.UTF-8
- name: JAVA_HOME
value: /usr/lib/jvm/java-1.8-openjdk/jre
- name: profiler.micro.service.tag.trace.enable
value: "true"
image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:0.1-SNAPSHOT
imagePullPolicy: Always
name: spring-cloud-a-gray
ports:
- containerPort: 20001
protocol: TCP
resources:
requests:
cpu: 250m
memory: 512Mi
livenessProbe:
tcpSocket:
port: 20001
initialDelaySeconds: 10
periodSeconds: 30
# B应用base版本,关闭无损下线能力。
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: spring-cloud-b
name: spring-cloud-b
spec:
replicas: 2
selector:
matchLabels:
app: spring-cloud-b
strategy:
template:
metadata:
labels:
app: spring-cloud-b
msePilotCreateAppName: spring-cloud-b
msePilotAutoEnable: "on"
spec:
containers:
- env:
- name: LANG
value: C.UTF-8
- name: JAVA_HOME
value: /usr/lib/jvm/java-1.8-openjdk/jre
- name: micro.service.shutdown.server.enable
value: "false"
- name: profiler.micro.service.http.server.enable
value: "false"
image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:0.1-SNAPSHOT
imagePullPolicy: Always
name: spring-cloud-b
ports:
- containerPort: 8080
protocol: TCP
resources:
requests:
cpu: 250m
memory: 512Mi
livenessProbe:
tcpSocket:
port: 20002
initialDelaySeconds: 10
periodSeconds: 30
# B应用gray版本,默认开启无损下线功能。
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: spring-cloud-b-gray
name: spring-cloud-b-gray
spec:
replicas: 2
selector:
matchLabels:
app: spring-cloud-b-gray
template:
metadata:
annotations:
alicloud.service.tag: gray
labels:
app: spring-cloud-b-gray
msePilotCreateAppName: spring-cloud-b
msePilotAutoEnable: "on"
spec:
containers:
- env:
- name: LANG
value: C.UTF-8
- name: JAVA_HOME
value: /usr/lib/jvm/java-1.8-openjdk/jre
image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:0.1-SNAPSHOT
imagePullPolicy: Always
name: spring-cloud-b-gray
ports:
- containerPort: 8080
protocol: TCP
resources:
requests:
cpu: 250m
memory: 512Mi
lifecycle:
preStop:
exec:
command:
- /bin/sh
- '-c'
- >-
wget http://127.0.0.1:54199/offline 2>/tmp/null;sleep
30;exit 0
livenessProbe:
tcpSocket:
port: 20002
initialDelaySeconds: 10
periodSeconds: 30
# C应用base版本
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: spring-cloud-c
name: spring-cloud-c
spec:
replicas: 2
selector:
matchLabels:
app: spring-cloud-c
template:
metadata:
labels:
app: spring-cloud-c
msePilotCreateAppName: spring-cloud-c
msePilotAutoEnable: "on"
spec:
containers:
- env:
- name: LANG
value: C.UTF-8
- name: JAVA_HOME
value: /usr/lib/jvm/java-1.8-openjdk/jre
image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:0.1-SNAPSHOT
imagePullPolicy: Always
name: spring-cloud-c
ports:
- containerPort: 8080
protocol: TCP
resources:
requests:
cpu: 250m
memory: 512Mi
livenessProbe:
tcpSocket:
port: 20003
initialDelaySeconds: 10
periodSeconds: 30
# HPA配置
---
apiVersion: autoscaling.alibabacloud.com/v1beta1
kind: CronHorizontalPodAutoscaler
metadata:
labels:
controller-tools.k8s.io: "1.0"
name: spring-cloud-b
spec:
scaleTargetRef:
apiVersion: apps/v1beta2
kind: Deployment
name: spring-cloud-b
jobs:
- name: "scale-down"
schedule: "0 0/5 * * * *"
targetSize: 1
- name: "scale-up"
schedule: "10 0/5 * * * *"
targetSize: 2
---
apiVersion: autoscaling.alibabacloud.com/v1beta1
kind: CronHorizontalPodAutoscaler
metadata:
labels:
controller-tools.k8s.io: "1.0"
name: spring-cloud-b-gray
spec:
scaleTargetRef:
apiVersion: apps/v1beta2
kind: Deployment
name: spring-cloud-b-gray
jobs:
- name: "scale-down"
schedule: "0 0/5 * * * *"
targetSize: 1
- name: "scale-up"
schedule: "10 0/5 * * * *"
targetSize: 2
---
apiVersion: autoscaling.alibabacloud.com/v1beta1
kind: CronHorizontalPodAutoscaler
metadata:
labels:
controller-tools.k8s.io: "1.0"
name: spring-cloud-c
spec:
scaleTargetRef:
apiVersion: apps/v1beta2
kind: Deployment
name: spring-cloud-c
jobs:
- name: "scale-down"
schedule: "0 2/5 * * * *"
targetSize: 1
- name: "scale-up"
schedule: "10 2/5 * * * *"
targetSize: 2
# Zuul网关开启SLB暴露展示页面
---
apiVersion: v1
kind: Service
metadata:
name: zuul-slb
spec:
ports:
- port: 80
protocol: TCP
targetPort: 20000
selector:
app: spring-cloud-zuul
type: ClusterIP
# A应用暴露K8s Service
---
apiVersion: v1
kind: Service
metadata:
name: spring-cloud-a-base
spec:
ports:
- name: http
port: 20001
protocol: TCP
targetPort: 20001
selector:
app: spring-cloud-a
---
apiVersion: v1
kind: Service
metadata:
name: spring-cloud-a-gray
spec:
ports:
- name: http
port: 20001
protocol: TCP
targetPort: 20001
selector:
app: spring-cloud-a-gray
# Nacos Server SLB Service配置
---
apiVersion: v1
kind: Service
metadata:
name: nacos-slb
spec:
ports:
- port: 8848
protocol: TCP
targetPort: 8848
selector:
app: nacos-server
type: LoadBalancer
- 执行以下命令:
kubectl apply -f mse-demo.yaml
预期输出:
deployment.apps/nacos-server created
service/nacos-server created
deployment.apps/spring-cloud-zuul created
deployment.apps/spring-cloud-a created
deployment.apps/spring-cloud-a-gray created
deployment.apps/spring-cloud-b created
deployment.apps/spring-cloud-b-gray created
deployment.apps/spring-cloud-c created
cronhorizontalpodautoscaler.autoscaling.alibabacloud.com/spring-cloud-b created
cronhorizontalpodautoscaler.autoscaling.alibabacloud.com/spring-cloud-b-gray created
cronhorizontalpodautoscaler.autoscaling.alibabacloud.com/spring-cloud-c created
service/zuul-slb created
service/spring-cloud-a-base created
service/spring-cloud-a-gray created
service/nacos-slb created
结果验证:无损下线功能
由于spring-cloud-b应用和spring-cloud-b-gray应用均开启了定时HPA,模拟每5分钟进行一次定时的扩缩容。
- spring-cloud-b应用:

- spring-cloud-b-gray应用:

- 登录MSE治理中心控制台。
- 在顶部菜单栏选择地域。
- 在左侧导航栏选择。
- 在应用列表列单击spring-cloud-a应用名称,右侧显示该应用相关的可观测数据。

从数据中可以看出:
- spring-cloud-a-gray版本的流量在Pod扩缩容的过程中请求错误数为0,无流量损失。
- spring-cloud-a版本由于关闭了无损下线功能,在Pod扩缩容的过程中有20个从spring-cloud-a发到spring-cloud-b的请求出现报错,发生了请求流量损耗。
结果验证:服务预热功能
在spring-cloud-c应用开启了定时HPA模拟应用启动的过程,每隔5分钟做一次伸缩,在第2分钟第0秒缩容到1个节点,在第2分钟第10秒扩容到2个节点。

- 登录MSE治理中心控制台。
- 在顶部菜单栏选择地域。
- 在左侧导航栏选择。
- 在应用列表列单击spring-cloud-c应用名称,在应用无损上线配置区域打开预热开关,预热时长设置为120秒。具体操作,请参见小流量预热服务。
观察节点的流量,发现节点流量缓慢上升,并且能看到节点的预热开始和结束时间,以及相关的事件。

从数据可以看出,开启预热功能的应用重启后的流量会随时间缓慢增加,在一些应用启动过程中需要预建连接池和缓存等资源的慢启动场景,开启服务预热能有效保护应用启动过程中缓存资源有序创建保障应用安全启动并做到流量无损。