To avoid security and stability risks associated with outdated cluster versions and to access the latest Kubernetes capabilities and technical support, upgrade your cluster version in a timely manner. ACK pre-checks, configurable upgrade policies and pacing, and upgrade progress monitoring help you complete the cluster upgrade smoothly.
Why you should upgrade
ACK lets you create clusters using the three latest minor versions of Kubernetes. For example, if ACK supports Kubernetes 1.31, 1.32, and 1.33, you can no longer create clusters that use version 1.30. You also cannot create clusters that use expired patch versions. For more information, see the version guide.
Expired cluster versions pose security and stability risks. After a version expires, you no longer receive new Kubernetes features, bug fixes, or timely technical support, leaving your cluster exposed to unpatched vulnerabilities.
Proactively upgrade your clusters in a timely manner to obtain the latest features and bug fixes and to receive more timely technical support.
Important When you upgrade a cluster, ACK runs pre-checks. However, these checks cannot detect all incompatible feature configurations or APIs. Under the shared responsibility model, you are responsible for staying informed about version releases by reading help documents and monitoring console and internal messages. Before you upgrade a cluster, review the upgrade considerations for the target version.
Upgrade impact
ACK provides phased and batched upgrade policies to ensure the continuity and stability of your services during a cluster upgrade.
Item | Description |
Pre-check | The pre-check identifies potential compatibility risks in the cluster, such as deprecated APIs, incompatible add-on versions, and issues with node or disk statuses, and provides suggestions on how to fix them. The pre-check result does not affect the workloads that run in the cluster. |
Control plane upgrade | The upgrade does not affect running applications in most cases. If an application is highly dependent on the API server, it may need to retry connections because of brief disconnections. ACK managed clusters: The API server is managed by ACK and is restarted in a rolling manner during the upgrade. ACK dedicated clusters: During the upgrade, ACK performs an in-place upgrade on each master node one by one.
Duration: ACK managed clusters and ACK serverless clusters, the upgrade is managed by ACK and takes approximately 5 minutes. For ACK dedicated clusters, the master nodes are upgraded one by one in sequence. Each node takes approximately 8 minutes. |
Node pool upgrade | Nodes are upgraded in batches to ensure service continuity. Customize the batch upgrade policy, such as the maximum number of nodes per batch and the interval between batches, to control the impact on your services.
In-place upgrade: Disks are not replaced and nodes are not re-initialized. Application pods continue to run, and services are not interrupted. Disk replacement upgrade: The node is drained, the system disk is replaced, and the node is re-initialized. Take note of the following items: ACK drains the node and evicts the pods on the node to other available nodes in accordance with the PDB. To ensure high availability, deploy multiple replicas across multiple nodes. Configure a
Pod Disruption Budget (PDB) for critical services to control the number of pods that can be disrupted at the same time.
ACK re-initializes the node based on the current configurations of the node pool, such as the node logon method, OS image, and container runtime version. Normally, you must update the node pool configurations by editing the node pool. If you modified the node in other ways, the changes are overwritten during the upgrade. If a pod on the node references a HostPath that points to the system disk, the data in the HostPath directory is lost after the disk replacement upgrade.
Duration:
The duration depends on how the nodes are batched. An in-place upgrade takes approximately 5 to 10 minutes per batch. A disk replacement upgrade without snapshots takes approximately 8 minutes. The specific duration is influenced by the node draining time.
To create snapshots, the upgrade starts after the snapshots are created. The time required to create snapshots depends on the data volume.
|
Upgrade process
An ACK cluster upgrade involves upgrading the control plane and node pools. Run a pre-check before you perform the upgrade, and schedule the upgrade during off-peak hours. After the control plane is upgraded, check the operational status of the cluster and then decide whether to upgrade the node pools.
Upgrade considerations
Control planes
You must upgrade the Kubernetes version of an ACK cluster sequentially through the supported versions. Rollbacks are not supported. To perform multiple upgrades, monitor your cluster's services for stability after each upgrade before you start the next one.
Review the version guide and the release notes of each version to understand the version details, deprecated APIs, and upgrade considerations of the target version. This avoids incompatibility issues caused by feature changes in later versions.
A control plane upgrade does not affect running applications. The API server is restarted in a rolling manner during the upgrade. If your application is highly dependent on the API server, it must be able to retry connections.
Kubernetes 1.24 and later do not support Docker as the built-in container runtime. When you upgrade a cluster from 1.22 to 1.24 or later, you must Migrate the node container runtime from Docker to containerd.
Avoid performing operations and maintenance (O&M) on the cluster during a control plane upgrade. Do not deploy or update your workloads during the upgrade. New pods may fail to obtain IP addresses while the API server is being restarted.
Node pools
-
Pre-upgrade preparation
-
The cluster upgrade uses yum to download the software packages required for the upgrade. If you have modified the network configuration of the nodes in your cluster or used a custom OS image, make sure that yum works properly on the nodes. You can run yum makecache to check.
-
Custom OS images are not strictly verified by ACK officially. ACK cannot fully guarantee that the upgrade succeeds.
-
If you have made configuration changes to the cluster, for example, enabled a SWAP partition, or modified the kubelet or runtime configuration by logging on to nodes instead of using the ACK console or the API (for the supported method, see Customize node pool kubelet configurations), the cluster upgrade may fail or your custom configurations may be overwritten.
-
In clusters running Kubernetes 1.18 or later, ACK configures Node resource reservation policy by default. If resource reservation is not configured for the cluster and the resource utilization of the nodes is high, evicted Pods may not be quickly scheduled after the upgrade. Reserve resources on the nodes. We recommend that the CPU utilization does not exceed 50% and the memory utilization does not exceed 70%.
-
In clusters running Kubernetes 1.24 or earlier, if the Pods of a workload are configured with only a startup probe, the Pods enter a brief NotReady state after kubelet restarts. We recommend that you deploy multiple replicas and spread the workload across multiple nodes to ensure that enough Pods remain available while a node restarts.
-
Reserve at least 20% of the disk space to prevent Pods from being evicted during the upgrade because the disk is too full.
-
Node pool upgrade constraints
-
During a node pool upgrade, only scale-out is supported. Scale-in is not supported.
-
If a node is an unmanaged node, that is, a worker node that is not managed by any node pool, see Migrate unmanaged nodes to a node pool to migrate it.
-
When nodes are upgraded by disk replacement, ACK reinitializes the nodes based on the current configuration of the node pool, such as the node logon method, labels, taints, OS image, and runtime version. Normally, the node pool configuration must be updated by Edit a node pool. If you have changed the nodes through other methods, these changes are overwritten during the upgrade.
-
If a Pod on a node references a HostPath volume that points to the system disk, the data in the HostPath directory is lost after the disk replacement upgrade.
When you upgrade a node pool in a cluster of version 1.31 or earlier, the process also upgrades the NVIDIA Device Plugin and resets any of its non-standard configurations.
-
node scaling and scheduling
-
If the node scaling feature is enabled for the cluster, to ensure that auto scaling is not affected, the cluster automatically updates cluster-autoscaler to the latest version after the cluster upgrade succeeds. After the cluster is upgraded, verify that the version of cluster-autoscaler is correct. For more information, see Enable node autoscaling.
-
During the cluster upgrade, nodes whose Scaling Mode is set to Swift may fail to complete the upgrade because the nodes are shut down. If any node remains un-upgraded after the upgrade ends because of the Swift mode, we recommend that you manually remove the node.
-
Networking and service availability
-
If a Pod accesses another Pod on the same node through the Server Load Balancer (SLB) address of a LoadBalancer Service, and the externalTrafficPolicy of the Service is set to Local, the two Pods may no longer reside on the same node after node rotation, which can cause network failures.
-
When nodes are upgraded by disk replacement, ACK drains the nodes and evicts the Pods on the nodes to other available nodes in accordance with the Pod Disruption Budget (PDB). To ensure high service availability, we recommend that you deploy multiple replicas and spread workloads across multiple nodes, and configure a PDB for critical workloads to control the number of Pods that can be disrupted at the same time.
The default timeout period for node draining is 30 minutes. If the Pod migration is not complete within the timeout period, ACK terminates the upgrade to ensure service stability.
Upgrade methods
In-place upgrade and disk replacement upgrade
For control plane upgrades, ACK manages the process for ACK managed clusters and ACK Serverless clusters. ACK dedicated clusters are upgraded using an in-place upgrade.
For a node pool upgrade, ACK offers two methods: in-place upgrade and disk replacement upgrade.
In-place upgrade: Upgrades are performed directly on existing nodes without replacing the system disk or re-initializing the node. The original node data is not affected. ECS instance-related configurations, such as IP addresses and disk mounts, remain unchanged. However, configurations of ACK-managed components, such as containerd and kubelet, may be adjusted based on differences between component versions.
To customize containerd or kubelet configurations, see Customize node pool kubelet configurations and Customize containerd configurations for a node pool.
Disk replacement upgrade: This method replaces the system disk and re-initializes the node. While IP addresses and data disk mounts remain unchanged, all data on the system disk is deleted. Enable disk snapshots and back up the system disk before proceeding.
Data disks attached to the node are not affected.
After a disk replacement upgrade, the default logon password of the ECS instance becomes invalid. Reset the password.
Special cases
A disk replacement upgrade is required in the following scenarios:
Alternative: rolling upgrade through a new node pool
Instead of upgrading existing nodes, perform a rolling upgrade by creating a new node pool with the target configuration. Gradually migrate applications by setting the old node pool to unschedulable or updating workload scheduling. Once the migration is verified, delete the old node pool.
You are billed for both node pools while they coexist; delete the old pool promptly after migration to manage costs.
FAQ
How do I manually upgrade my cluster?
For more information, see Manually upgrade a cluster. First, complete the pre-check. Then, perform the upgrade and verify that the result meets your expectations.
What are the best practices for upgrading a cluster?
Maintain a regular upgrade frequency: Monitor the version guide
, help documents, emails, and text messages. Upgrade promptly to ensure continued support and security.
Create an upgrade plan: A cluster upgrade involves more than a version number change. It also involves deprecated APIs, deprecated features, and major changes. Create a detailed upgrade plan based on cluster size and business needs, and reserve a maintenance window. Upgrade during off-peak hours. Validate the upgrade in a test environment before you apply it to production. After the upgrade, verify and monitor your workloads to confirm that the upgrade does not affect your business.
Use automatic upgrades: Enable the automatic cluster upgrade feature. ACK generates an upgrade plan in advance, triggers a pre-check, and upgrades the cluster within the specified maintenance window. This reduces the O&M workload of version management.
Alternatively, create an ACK Auto Mode cluster
. The cluster version is automatically upgraded by ACK.
How long does a cluster upgrade take?
Control plane: For ACK managed clusters and ACK Serverless clusters, the upgrade is managed by ACK and takes about 5 minutes. For an ACK dedicated cluster, the master nodes are upgraded sequentially, which takes about 8 minutes per node.
Node pool: The duration depends on the node batching configuration. An in-place upgrade takes about 5 to 10 minutes per batch. A disk replacement upgrade without snapshots takes about 8 minutes per batch, but the actual duration is affected by the node draining process. If you choose to create snapshots, the upgrade process waits for the snapshots to be created. The time required to create a snapshot depends on the amount of data.
Can I stay on one version forever and not upgrade my cluster?
No, you cannot. Potential security risks in outdated versions can affect not only your clusters but also the overall security of Alibaba Cloud. ACK does not allow clusters to remain in an outdated state for an extended period and performs forced upgrades to bring them to a secure and stable version.
Upgrade your cluster version promptly. For more information, see Manually upgrade a cluster. This lets you benefit from the latest features and receive better technical support from ACK. Before you upgrade, see the release notes for the target version to understand its feature changes and important notes. Enable the automatic cluster upgrade feature to ensure your cluster is upgraded automatically and periodically.
Does ACK support upgrading across multiple versions?
My cluster version is very old. How can I upgrade it quickly?
Use one of the following solutions.
Solution 1: Upgrade the cluster one minor version at a time. After each upgrade, check whether your business applications in the cluster run as expected before you proceed with the next upgrade. For more information, see Manually upgrade a cluster.
Solution 2: Create a cluster that runs the latest version, gradually migrate your applications to the new cluster, and then delete the old cluster. For information about how to create and configure a cluster, see Create an ACK managed cluster.
How do I switch from Docker to containerd when upgrading a cluster from version 1.22 to 1.24?
ACK no longer supports Docker as the built-in container runtime in version 1.24 and later. You must migrate the container runtime of your nodes from Docker to containerd.
Switch the runtime in the original node pool by using the node pool upgrade feature, or create a new containerd node pool and migrate your workloads. For more information, see Migrate the container runtime of nodes from Docker to containerd.
How does ACK ensure stability during a cluster upgrade?
An ACK cluster consists of a control plane and node pools. For both the control plane and node pools, the pre-check results do not affect running applications, and the console provides repair suggestions if a check fails.
Control plane upgrade: ACK provides a pre-check feature that inspects deprecated APIs, add-on compatibility, feature configuration compatibility, and control plane components. For more information, see Manually upgrade a cluster.
Node pool upgrade: A node pool upgrade includes upgrading kubelet and containerd. ACK provides a pre-check feature that inspects the node status, system resources, disk status, and network environment.
Also configure an upgrade policy to control the upgrade pace. For example, specify the nodes to upgrade, set the maximum number of nodes that can be upgraded in each batch, and configure an upgrade pause policy. If the system disks of your nodes contain important business data, you can also create snapshots for the nodes before you upgrade the node pool. For more information, see Upgrade a node pool.
What should I know before upgrading a cluster?
You cannot roll back a cluster upgrade. First upgrade a test environment and verify the result, then upgrade the production environment. Alternatively, upgrade specific nodes first for verification during the upgrade process.
The supported component versions, features, and deprecated features vary with the Kubernetes version. For more information, see the release notes for different versions.
Review the notes on control plane upgrades.
Review the notes on node pool upgrades.
Does a forced upgrade affect my workloads?
A forced upgrade is an in-place upgrade. It does not replace disks or change the operating system or container runtime. Before ACK forcibly upgrades a cluster, ACK notifies you at least one month in advance by text message, email, and internal message.
The following clusters are not forcibly upgraded:
Can clusters with expired versions still be used normally?
Yes. However, clusters that use expired versions are exposed to security and stability risks. Upgrade the cluster to a maintained version as soon as possible.
Because ACK clusters use a managed architecture, these security risks not only affect your cluster but may also impact the overall security of Alibaba Cloud. Therefore, ACK does not allow clusters to remain in an outdated state for an extended period and performs a forced upgrade to a secure and stable version. For more information, see Mandatory upgrade of outdated versions.
Is version rollback supported after a cluster upgrade?
No. Rollbacks are not supported for the control plane, kubelet, or container runtime versions after an upgrade.
During a node pool upgrade, if important service data is stored on a node's system disk, create a snapshot for the node before the upgrade to back up and restore node data.
Which operation should I perform first if I need to upgrade a cluster and migrate it to an ACK Pro managed cluster?
Complete the cluster migration first. After you verify that your services run normally, start the cluster upgrade.
What do I do if the pre-check reports deprecated APIs?
When you upgrade a cluster that runs Kubernetes 1.20 or later, ACK checks whether the current version uses deprecated APIs. These results do not block the upgrade and are informational only. Fix the reported issues before the upgrade to avoid affecting normal cluster operation on the new version.
How do I resolve a "component version too low" warning in pre-checks?
Except for the control plane components and some core components (such as kube-proxy), ACK does not upgrade other cluster add-ons. View the add-ons that need to be upgraded on the Add-ons page in the console. Refer to Component overview and release notes to learn about version compatibility and upgrade requirements. Then upgrade these add-ons during off-peak hours.
Is it normal for nodes to raise alerts during an upgrade?
The following expected temporary alerts may occur during an upgrade. No manual intervention is required:
Alerts about failed synchronization of ConfigMaps, Secrets, and ServiceAccounts during kubelet restarts.
FailedMount alerts during disk replacement upgrades (the nodes are being rebuilt).
Brief periods in which nodes are in the NotReady state.
These alerts automatically recover after the upgrade is complete, and usually last a few minutes.
Can I upgrade a cluster if the vSwitch has no available IP addresses?
A disk replacement upgrade requires available IP addresses to create the temporary ECS network for the new system disk. If the vSwitch does not have enough available IP addresses, Secondary CIDR blocks, or add a vSwitch that has more available IP addresses to the node pool before you upgrade.
How do I resolve an upgrade failure with the error "the aliyun service is not running on the instance"?
Cloud Assistant is unavailable, which causes the upgrade command to fail.
Start or stop Cloud Assistant, and then try the cluster upgrade again. For more information, see Start, stop, or uninstall Cloud Assistant Agent.
How do I resolve a "PLEG not healthy" error on a node?
This error indicates that the container or container runtime is not responding. Restart the node and then try the upgrade again.
What should I do if I get an invalid object doesn't have additional properties error when upgrading a cluster?
After you upgrade a cluster, you must also upgrade your local kubectl version. If you do not upgrade it promptly, you may encounter errors such as invalid object doesn't have additional properties when you use the local kubectl because its version differs from the cluster's API server version. For more information about how to install or upgrade kubectl, see Install and Set Up kubectl.