Implement a canary release using NGINX Ingress and Apsara Devops AppStack
This topic describes how to configure NGINX Ingress and Apsara Devops AppStack to implement a canary release.
Background information
Canary releases reduce deployment risks and improve service stability. They are ideal for rapid software development iterations. Compared to Kubernetes rolling deployments, canary releases based on traffic features are more precise and less risky. In cloud-native scenarios, canary releases that use NGINX Ingress are common. This method directs traffic at the ingress point. It routes canary traffic to the new service version for validation. After successful validation, the new version is fully deployed. If validation fails, you can roll back the changes promptly to ensure uninterrupted user services.
Basic principles
For a typical web service, the basic logic of an Ingress canary release is as follows: Requests to the same ingress (HOST) are partially routed to the canary service based on specific features or a traffic ratio. You then monitor and validate the canary traffic to determine if the canary service meets the standards for production release.
The following figure shows the architecture of an Ingress canary release on Kubernetes based on the traffic path.
Traffic enters through the HOST. Based on the NGINX Ingress traffic identifier, which is defined in the Ingress annotation, requests that match the canary identifier are routed to the service in the canary environment. The requests then reach the workload pods.
The services and deployments within each environment are not aware of the canary identifier. Internal remote procedure call (RPC) traffic usually does not carry the canary identifier. Therefore, this architecture primarily addresses canary releases for external traffic. It is suitable for scenarios with a small number of services and simple service-to-service calls.
Procedure
This section describes how to perform an Ingress canary release for an application on an Alibaba Cloud Container Service for Kubernetes (ACK) cluster using Apsara Devops AppStack.
First, import an ACK cluster and create an application.
Environment management
Create an environment
On the homepage of Apsara Devops AppStack, go to the target application. Create a canary environment and a production environment. Both environments share the same Kubernetes cluster. Name the canary environment grey and the production environment ack-prod.
Configure the orchestration
Click to define the deployment orchestration. The key to enabling a canary release is the canary annotation in the Ingress. Check your NGINX Ingress controller version to determine which annotations are supported. The NGINX Ingress controller on the test cluster is version 0.30. It uses a header to identify canary traffic.
To enable the canary routing configuration only in the canary environment, use a conditional statement in the orchestration template. The following code provides an example:
--- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: {{ .AppStack.appName }}-{{ .AppStack.envName }} namespace: {{ .Values.namespace }} {{ if eq .AppStack.envName "grey" }} annotations: # Enable Canary. nginx.ingress.kubernetes.io/canary: "true" # The request header is _env. nginx.ingress.kubernetes.io/canary-by-header: "_env" # Requests are routed to the new service version only when the value of the _env request header is grey. nginx.ingress.kubernetes.io/canary-by-header-value: "grey" {{ end }} spec: rules: - host: {{ .Values.host }} http: paths: - path: / pathType: Prefix backend: service: name: {{ .AppStack.appName }}-{{ .AppStack.envName }} port: number: 80In the Ingress orchestration, you can see that the canary annotation is enabled only for the environment named "grey". The canary identifier is set using the header tag _env: grey.
In the Ingress rules configuration, the host variable specifies the associated HOST for routing. The ACK canary environment and the ACK production environment are associated with the canary environment variable group and the production environment variable group, respectively. The host variable in both groups has the same value.
Example:
# namespace = demo-pre # host = go.demo.prod# namespace = demo-prod # host = go.demo.prodIn addition to the Ingress, you also need to configure the corresponding Service and Deployment.
--- apiVersion: v1 kind: Service metadata: name: {{ .AppStack.appName }}-{{ .AppStack.envName }} namespace: {{ .Values.namespace }} spec: selector: run: {{ .AppStack.appName }}-{{ .AppStack.envName }} ports: - protocol: TCP port: 80 targetPort: 8080--- apiVersion: apps/v1 kind: Deployment metadata: name: {{ .AppStack.appName }}-{{ .AppStack.envName }} labels: run: {{ .AppStack.appName }}-{{ .AppStack.envName }} namespace: {{ .Values.namespace }} spec: replicas: {{ .Values.replicas }} selector: matchLabels: run: {{ .AppStack.appName }}-{{ .AppStack.envName }} template: metadata: labels: run: {{ .AppStack.appName }}-{{ .AppStack.envName }} spec: containers: - name: main image: {{ .AppStack.image.backend }} ports: - containerPort: 8080 resources: limits: cpu: {{ .Values.cpuLimit }} memory: {{ .Values.memoryLimit }} requests: cpu: {{ .Values.cpuRequest }} memory: {{ .Values.memoryRequest }}
Prepare the sample code and associate it with the application
Download the sample code from atomgit at the following path: https://atomgit.com/feiyuw/demo-go-echo.git
Import the code into Apsara Devops Codeup and then associate it with the application.
Define the canary release flow
Next, use the development flow in Apsara Devops AppStack to define the canary release process. The following figure shows an example.
In this flow, each execution pulls code from the master branch to build an image. After the build is complete, the process is submitted for O&M approval. The deployment process begins after approval.
The deployment process includes four steps: canary deployment, canary validation, production deployment, and canary cleanup.
The canary deployment step updates the ACK canary environment with the newly built image. At this point, you can perform canary validation by including the _env:grey header in your requests.
Canary validation is a manual gate for observing and validating the canary environment. If validation passes, the production environment is deployed automatically. If validation fails, the production deployment is skipped, and the canary cleanup is executed.
The production deployment step updates the ACK production environment with the image. The new service version is now visible to regular users. Note: To reduce risk, the production deployment policy is set to deploy in batches and pause after the first batch. This ensures a gradual rollout to the production environment and allows for a quick rollback if needed.
The canary cleanup step removes resources from the canary environment. This saves resources and prevents any impact on the production environment if canary validation fails.
In the settings for the application on Apsara Devops AppStack, configure the development flow. If you only want to test the canary release feature, you can configure only the production stage.
You can define the flow using the following pipeline YAML. Replace the registry address and service connection in the acr_docker_build_step with your actual values. Also, replace the userId in grey_validate and ops_validate with your Alibaba Cloud account ID.
---
stages:
build:
name: "Build"
jobs:
go_build:
name: "Go Image Build"
steps:
golang_build_step:
name: "Golang Build"
step: "GolangBuild"
with:
goVersion: "1.20.x"
run: |
export GOPROXY=https://goproxy.cn
make build
upload_step:
step: "ArtifactUpload"
name: "Upload Build Output"
with:
serviceConnection: "wtdbdh89rrfdsod6"
repo: "flow_generic_repo"
artifact: "demo-go-echo"
version: "prod-${CI_COMMIT_ID}.${DATETIME}"
filePath:
- "demo-go-echo"
- "deploy.sh"
acr_docker_build_step:
name: "Build and push the image to an individual edition of Alibaba Cloud Container Registry"
step: "ACRDockerBuild"
with:
artifact: "image"
dockerfilePath: "Dockerfile"
dockerRegistry: "registry.cn-zhangjiakou.aliyuncs.com/docker007/demo-go-echo"
dockerTag: "prod-${CI_COMMIT_ID}.${DATETIME}"
region: "cn-zhangjiakou"
serviceConnection: "<connectionId>"
approve:
name: "Deployment Approval"
jobs:
ops_validate:
name: "O&M Approval"
component: "ManualValidate"
with:
validatorType: "users"
validators:
- <userId>
grey:
name: "Canary Validation"
jobs:
grey_deploy_job:
name: "ACK Canary Deployment"
component: "AppStackFlowDeploy"
with:
application: "demo-go-echo"
environment: "grey"
artifacts:
- label: "backend"
value: "$[stages.build.go_build.acr_docker_build_step.artifacts.image.dockerUrl]"
autoDeploy: true
grey_validate:
name: "Canary Validation"
component: "ManualValidate"
needs:
- "grey_deploy_job"
with:
validatorType: "users"
validators:
- <userId>
deploy:
name: "Deploy"
jobs:
ack_deploy_job:
name: "ACK Production Deployment"
component: "AppStackFlowDeploy"
condition: |
succeed('grey.grey_validate')
with:
application: "demo-go-echo"
environment: "ack-prod"
artifacts:
- label: "backend"
value: "$[stages.build.go_build.acr_docker_build_step.artifacts.image.dockerUrl]"
autoDeploy: true
cleanup:
name: "Clean Up Environment"
jobs:
cleanup_grey_env_job:
name: "Clean Up Canary Environment"
component: "AppStackCleanEnv"
needs:
- "grey.grey_validate"
- "deploy.ack_deploy_job"
condition: |
failed('grey.grey_validate') || succeed('deploy.ack_deploy_job')
with:
application: "demo-go-echo"
environment: "grey"
deleteEnv: "cleanEnv"
Validate the canary release flow
Assume the development flow contains only the production stage. Run the production stage once to deploy the application to the canary and production environments. Note: For the first deployment to an environment, you may need to create a deployment ticket manually.
Modify the code by changing the version number in routes.go to a new value. Run the production stage again until it reaches the canary validation step. At this point, the production and canary environments are running different versions.
Run kubectl get ing -A to retrieve the Ingress egress IP address. Bind the IP address to go.demo.prod in your local /etc/hosts file. For example:
127.0.0.1 go.demo.prod # Replace 127.0.0.1 with the correct egress IP addressOpen a terminal and send a request to the /version endpoint using httpie or curl. The following example uses httpie:
http -v http://go.demo.prod/version # Request the production environment
http -v http://go.demo.prod/version _env:grey # Request the canary environmentFAQ
If I have a canary environment, do I still need to deploy to the production environment in batches?
Yes. We recommend that you use batch deployments for the production environment. Although the canary environment passed validation, the production environment can be affected by factors such as data volume. A full-scale release is not recommended. Batch deployments limit the risk to a small scope and help prevent major failures.
How can I integrate configuration changes and data changes into the development flow?
You can add configuration changes and data changes as steps in the development stage of the pipeline YAML. We recommend that you perform data changes before application deployment and configuration changes after deployment. Also, if you use a configuration center such as Nacos for the canary environment on Kubernetes, you should have a corresponding canary namespace. This prevents direct modifications to the production configuration.
If a canary environment contains multiple applications, how can I ensure that internal service calls are also routed within the canary environment?
For a complete solution, consider using products such as Microservices Engine (MSE).
If you have a small number of applications, a simple call chain, and can use a basic Kubernetes-based solution, you can define a canary environment for each application. These environments can share the same cluster and namespace. Applications can call each other through Services. Because of namespace fencing, applications in the canary environment will only call the canary versions of other applications in the same namespace.
This solution requires the long-term availability of each application's canary environment. Therefore, you must remove the final cleanup step from the development flow.
How can I associate other types of releases, such as Function Compute, in the flow?
You can orchestrate the relevant Function Compute steps into the development flow pipeline YAML. This enables filter interaction between them.