Auto upgrade of clusters and components
An ACK Auto Mode cluster must keep upgrading its Kubernetes version and updating its components to stay secure and stable. The automatic upgrade feature completes Kubernetes version upgrades, data plane component upgrades (kubelet and container runtime), and cluster core component (add-on) upgrades within the maintenance window that you specify. By combining an upgrade channel with a maintenance window policy, you keep the cluster on a supported, secure version with minimal manual work.
Key concepts
Concept | Description |
Upgrade channel | The policy that controls the step size of automatic cluster version upgrades. Three channels are available: |
Maintenance window | The time range in which automated operations and maintenance (O&M) tasks can run. Tasks are neither triggered nor interrupted outside the window. |
Execution plan | Represents one specific O&M action. Each plan has a complete state lifecycle. |
Control plane | The master components of the cluster, including the Kubernetes API server, controller manager, and scheduler. |
Data plane | The kubelet and the container runtime (containerd) on nodes. |
Automatic Kubernetes version upgrades
Automatic upgrades in an Auto Mode cluster differ from those in a standard ACK managed cluster in the following ways:
Automatic version upgrades are enabled by default when an Auto Mode cluster is created. You do not need to enable them manually.
Component automatic upgrades are mandatory for Auto Mode clusters and cannot be disabled.
Automatic upgrades cover three layers: the control plane, the data plane (kubelet and containerd), and system components (add-ons). All upgrades are driven by a unified maintenance window and execution plan.
Cluster upgrade channels
Auto Mode manages cluster version upgrades through upgrade channels. Choose the channel that matches how much stability your workloads need and how quickly you want new features:
Channel | Upgrade behavior | Scenarios |
Latest patch version ( | When a patch version is available for the current minor version, the cluster is automatically upgraded to that patch version, for example from 1.34.3-aliyun.1 to 1.34.6-aliyun.1. No breaking changes are involved. | You prioritize stability and accept only security and bug fixes. |
Second-latest minor version ( | The cluster is automatically upgraded to the latest patch version of the second-latest minor version. A new Kubernetes version may change APIs and features, but its stability has already been widely validated. | You want to balance stability with new features. |
Latest minor version ( | The cluster is automatically upgraded to the latest patch version of the latest minor version, so that you get new Kubernetes community features sooner. | You want to try new version capabilities as soon as possible. New minor versions may change APIs and features. |
Upgrade layers and order
Automatic cluster version upgrades cover two layers, which run in a strict order:
Data plane kubelet upgrade: upgrades the kubelet and container runtime on the nodes in a node pool to the same version as the control plane.
Control plane upgrade: upgrades the Kubernetes version of the cluster control plane.
Before the control plane is upgraded, the system checks that the kubelet and containerd versions on the data plane meet the compatibility requirements of the target version. For example, an upgrade to Kubernetes 1.24 requires a switch to the containerd runtime, and an upgrade to Kubernetes 1.26 requires containerd 1.6 or later. If the data plane component versions do not meet the prerequisites for a control plane upgrade, the system does not trigger a control plane upgrade plan.
Version path calculation
Based on the upgrade channel configured for the cluster, the system starts from the current version and calculates the next upgradable version step by step to generate a deterministic upgrade path. The calculation follows these rules:
Step constraint: only one minor version is advanced at a time. Upgrades that skip minor versions are not allowed.
Target version selection: for each minor version, the latest patch version of the channel is anchored automatically.
Data plane baseline: the start version of the path calculation is the lowest kubelet version number among all nodes in the node pool.
The following example shows the stable channel, assuming that the current cluster version is 1.34:
Current version 1.34.x → 1.35.x (latest patch) → 1.36.x (latest patch) → ... → final stable versionNode upgrade policy
The kubelet in a node pool is upgraded in rolling batches to keep your workloads running:
Parallelism (
MaxParallelism): one node per batch by default.Node drain (Drain): before a node is upgraded, the node is cordoned so that no new pods are scheduled to it, and its pods are evicted.
Skipped pods: pods managed by
DaemonSets and static image pods are skipped automatically.Local storage: pods that use local storage (
emptyDir) are handled automatically.Termination grace period: the pod termination grace period is 90 seconds by default.
Batch interval: you can configure how long the system waits between batches, which gives you time to observe and validate the state during the upgrade.
By default, all intelligent managed node pools in an Auto Mode cluster take part in automatic upgrades. The upgrade policy is driven by the cluster-level configuration. You cannot disable automatic kubelet upgrades for an individual node pool.
Lingjun node pools and hybrid node pools do not support automatic kubelet upgrades.
Configure automatic upgrades
Automatic version upgrades are already enabled when an Auto Mode cluster is created. You only need to configure the upgrade channel and the maintenance window. The maintenance window covers the automatic upgrade tasks of the cluster and its components.
Automatic upgrades take effect only when the maintenance window is enabled. Configure the window so that it covers your off-peak hours.
Log on to the Container Service Management Console.
On the Clusters page, click the cluster that you want to manage.
In the left-side navigation pane, click Cluster Information, and then click the Basic Information tab to go to the cluster basic information page.
Next to Automatic Version Update, click Edit to configure the upgrade channel (
patch,stable, orrapid).Next to Maintenance Window, click Enabled to enable and configure the maintenance window.
After you complete the configuration, automatic upgrades of the cluster and its components run within the maintenance window.
Automatic component upgrades
Component automatic upgrades target the core system components of a cluster and upgrade them to the recommended version within the maintenance window. You configure the maintenance window as described in Configure automatic upgrades.
Supported components
The components that currently support automatic upgrades are listed by category.
Core components
Component | Description |
Connects Kubernetes to Alibaba Cloud infrastructure and manages load balancing (CLB and NLB) and cross-node network communication. | |
The node elastic scaling controller of an Auto Mode cluster. It scales nodes in and out automatically based on workload demand. |
Network
Component | Description |
The control plane component of the Terway network plug-in. It handles network policy delivery, IP resource allocation, and network topology management. | |
The Terway CNI network plug-in developed by Alibaba Cloud. It supports | |
The data plane component for the Terway ENI multi-IP mode. It allocates IP addresses for elastic network interfaces (ENI) and forwards container network traffic. |
Storage
Component | Description |
Manages the lifecycle of storage components and coordinates the installation, upgrade, and configuration of CSI plug-ins. | |
The CSI node plug-in. It mounts and unmounts persistent volumes (PV) such as disks, NAS file systems, and OSS buckets. | |
The CSI control plane plug-in. It supports dynamic volume provisioning and volume lifecycle management, and includes the managed version managed-csiprovisioner. | |
The container network file system controller. It provides lifecycle management and performance optimization for NAS volumes. |
Logging and monitoring
Component | Description |
Monitors abnormal events on cluster nodes. It detects node faults and reports events automatically. | |
Processes cost analysis data and provides data collection and computing support for the cluster cost insights feature. | |
Collects logs. It collects container standard output and file logs and delivers them to the destination. | |
Integrates with ARMS Prometheus monitoring and collects and monitors metrics for the cluster and its workloads. | |
Aggregates resource metrics and provides the Metrics API that supports Horizontal Pod Autoscaler (HPA) scaling. |
Security
Component | Description |
Inspects cluster security. It periodically scans cluster configurations and runtime security risks. | |
The RAM identity authentication component. It provides authentication and authorization for the cluster based on Alibaba Cloud RAM. |
GPU
Component | Description |
The NVIDIA GPU device plug-in. It registers the GPU devices on a node with Kubernetes and lets pods be scheduled to use them. |
Execution characteristics
Cluster-level operation: component upgrades apply to the whole cluster control plane.
Staggered scheduling: when several components in the same cluster need an upgrade, they are upgraded one after another within the same O&M window, with an interval of about one hour between components. This avoids the risk of simultaneous changes.
Auto Mode cluster constraint: component automatic upgrades are mandatory for Auto Mode clusters and cannot be disabled. For details, see Automatic Kubernetes version upgrades.
Execution plan lifecycle
Every automatic upgrade action is managed by an execution plan. The following figure shows how the state of an execution plan changes.
State | Description |
Scheduled | The plan is generated and waits for the maintenance window to start. |
Executing | The plan is performing the O&M operation. |
Executed | The O&M operation is complete. |
Canceled | The plan is canceled because the configuration was disabled, the cluster was deleted, the window was exceeded, or for a similar reason. |
Failed | The plan still fails after multiple retries and is archived. |
In the following situations, the related running plans are canceled automatically:
You disable cluster automatic upgrades or the managed configuration of a node pool.
The cluster is in a state other than running, such as being deleted or failed to delete.
A trial user runs out of CVE fix quota or has insufficient Cloud Security authorizations. In this case, an alert is generated. Pay attention to the notifications.
Execution flow
Plan generation: the system inspects all clusters, identifies the objects that need an upgrade, and generates plans.
Conflict scheduling: plans are deduplicated, throttled, and staggered to avoid concurrent execution of conflicting operations in the same cluster or node pool.
Window scheduling: only plans that can run within the maintenance window are selected and submitted one by one.
Execution and synchronization: the underlying O&M interface is called to run the plan, and task progress is synchronized to the plan state periodically.
Notification: a notification is pushed automatically when execution fails.
Safety mechanisms
Maintenance window constraint: all automated O&M operations run strictly within the window.
Concurrency control:
Multiple upgrade plans do not run concurrently in the same cluster.
Plans do not run concurrently in the same node pool.
Circuit-breaker protection: when a type of upgrade task fails consecutively and reaches the threshold, the system automatically breaks the circuit for that task type to keep the fault from spreading.
Pre-upgrade validation: before an upgrade, the system runs checks such as version consistency, node availability, and runtime compatibility. If a check fails, the upgrade is blocked and reported.
Cancelable: you can cancel an O&M plan.