DocsICP FilingConsole
Documentation
Home Page Container Service for Kubernetes ACK Edge Clusters User Guide Node pool management Cloud node pool management Node pool O&M Upgrade a node pool

Upgrade a node pool

Updated at:
Copy as MD
Product
My Favorites

After the control plane upgrade, complete the node pool upgrade during off-peak hours. ACK runs a pre-upgrade check to identify risk factors so that the kubelet and container runtime upgrade proceeds smoothly.

Considerations

  • Pre-upgrade preparation

    • The cluster upgrade uses yum to download the software packages required for the upgrade. If you have modified the network configuration of the nodes in your cluster or used a custom OS image, make sure that yum works properly on the nodes. You can run yum makecache to check.

    • Custom OS images are not strictly verified by ACK officially. ACK cannot fully guarantee that the upgrade succeeds.

    • If you have made configuration changes to the cluster, for example, enabled a SWAP partition, or modified the kubelet or runtime configuration by logging on to nodes instead of using the ACK console or the API (for the supported method, see Customize node pool kubelet configurations), the cluster upgrade may fail or your custom configurations may be overwritten.

    • In clusters running Kubernetes 1.18 or later, ACK configures Node resource reservation policy by default. If resource reservation is not configured for the cluster and the resource utilization of the nodes is high, evicted Pods may not be quickly scheduled after the upgrade. Reserve resources on the nodes. We recommend that the CPU utilization does not exceed 50% and the memory utilization does not exceed 70%.

    • In clusters running Kubernetes 1.24 or earlier, if the Pods of a workload are configured with only a startup probe, the Pods enter a brief NotReady state after kubelet restarts. We recommend that you deploy multiple replicas and spread the workload across multiple nodes to ensure that enough Pods remain available while a node restarts.

    • Reserve at least 20% of the disk space to prevent Pods from being evicted during the upgrade because the disk is too full.

  • Node pool upgrade constraints

    • During a node pool upgrade, only scale-out is supported. Scale-in is not supported.

    • If a node is an unmanaged node, that is, a worker node that is not managed by any node pool, see Migrate unmanaged nodes to a node pool to migrate it.

    • When nodes are upgraded by disk replacement, ACK reinitializes the nodes based on the current configuration of the node pool, such as the node logon method, labels, taints, OS image, and runtime version. Normally, the node pool configuration must be updated by Edit a node pool. If you have changed the nodes through other methods, these changes are overwritten during the upgrade.

    • If a Pod on a node references a HostPath volume that points to the system disk, the data in the HostPath directory is lost after the disk replacement upgrade.

    • When you upgrade a node pool in a cluster of version 1.31 or earlier, the process also upgrades the NVIDIA Device Plugin and resets any of its non-standard configurations.

  • node scaling and scheduling

    • If the node scaling feature is enabled for the cluster, to ensure that auto scaling is not affected, the cluster automatically updates cluster-autoscaler to the latest version after the cluster upgrade succeeds. After the cluster is upgraded, verify that the version of cluster-autoscaler is correct. For more information, see Enable node autoscaling.

    • During the cluster upgrade, nodes whose Scaling Mode is set to Swift may fail to complete the upgrade because the nodes are shut down. If any node remains un-upgraded after the upgrade ends because of the Swift mode, we recommend that you manually remove the node.

  • Networking and service availability

    • If a Pod accesses another Pod on the same node through the Server Load Balancer (SLB) address of a LoadBalancer Service, and the externalTrafficPolicy of the Service is set to Local, the two Pods may no longer reside on the same node after node rotation, which can cause network failures.

    • When nodes are upgraded by disk replacement, ACK drains the nodes and evicts the Pods on the nodes to other available nodes in accordance with the Pod Disruption Budget (PDB). To ensure high service availability, we recommend that you deploy multiple replicas and spread workloads across multiple nodes, and configure a PDB for critical workloads to control the number of Pods that can be disrupted at the same time.

      The default timeout period for node draining is 30 minutes. If the Pod migration is not complete within the timeout period, ACK terminates the upgrade to ensure service stability.

Feature description

A node pool upgrade includes a kubelet upgrade and a container runtime upgrade.

  • kubelet upgrade: upgrades the kubelet of the nodes in the node pool to the same version as the control plane. By default, the in-place upgrade method is used.

  • Container runtime upgrade: after a new version of the container runtime is released, you can upgrade the container runtime of the nodes to the latest version.

    • When you upgrade from Docker to containerd, the nodes in the node pool are upgraded by resetting the nodes through disk replacement (replacing the node system disk). The upgrade erases all content on the system disk. Before the disk replacement upgrade, make sure that you back up the important data on the system disk. For more information, see Migrate the node container runtime from Docker to containerd.

    • Except for ContainerOS nodes, when you upgrade from containerd to a later version of containerd, the in-place upgrade method is used by default. The /etc/containerd/config.toml on the node is replaced with the new version provided by ACK.

      Important

      • ContainerOS nodes do not support in-place upgrades of containerd. Only disk replacement upgrades are supported. For more information about the upgrade process, see Upgrade pre-3.4 ContainerOS to the latest version.

      • During the runtime upgrade, Pod probes and lifecycle hooks may fail, and Pods may be restarted in place.

    • In clusters running Kubernetes 1.24 or earlier, when you upgrade from Docker to a later version of Docker, the nodes in the node pool are upgraded by disk replacement by default. The upgrade erases all content on the system disk. Before the disk replacement upgrade, make sure that you back up the important data on the system disk.

Procedure

  1. Log on to the ACK console. In the left navigation pane, click Clusters.

  2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.

  3. In the Node Pools list, find the target node pool. In the Actions column, click image > Kubelet Update and configure the following parameters.

    Parameter

    Description

    Kubelet Update Information

    View the current kubelet version and select the target version to upgrade to.

    Runtime Update Information

    View the current container runtime version and select the target runtime version.

    • When you upgrade from Docker to containerd, the nodes in the node pool are upgraded by resetting the nodes through disk replacement (replacing the node system disk). The upgrade erases all content on the system disk. For more information, see Migrate the node container runtime from Docker to containerd.

    • If your cluster runs Kubernetes 1.22 and the installed containerd version is 1.6.34, container runtime upgrades are not supported.

    Update Nodes

    Specify the nodes to upgrade (all nodes or selected nodes).

    Upgrade Method

    Select an upgrade method. Valid options: In-place Upgrade and Upgrade by Replacing System Disk. For more information about the upgrade logic and process, see Reference information: in-place upgrade and disk replacement upgrade.

    • In-place Upgrade: Select this option to update the required components directly on the original node. An in-place upgrade does not replace the system disk or reinitialize the node, and the data on the original node is not affected.

    • Upgrade by Replacing System Disk: Select this option to reinitialize the node by replacing its system disk. The instance properties of the node, such as the node name, instance ID, and IP address, remain unchanged. However, all data on the system disk is deleted. Data disks attached to the node are not affected.

    Ignore Warnings

    Whether to proceed if the precheck reports warnings. For example, a pod uses a hostPath pointing to the system disk.

    Batch Update Policy

    Maximum Number of Nodes per Batch

    The upgrade process upgrades the nodes in batches based on the maximum number of nodes per batch that you set. The nodes within each batch are upgraded at the same time. For more information about the upgrade process, see Reference information: in-place upgrade and disk replacement upgrade below.

    Automatic Pause Policy

    Specifies when the upgrade pauses automatically.

    Interval Between Batches

    If the automatic pause policy is set to not pause automatically, you can specify whether to add a time interval between upgrade batches and the length of the interval. Valid values: 5 to 120 minutes.

    Auto Snapshot

    If important business data is stored on the system disks of your nodes, you can specify whether to create snapshots for the nodes before the node pool upgrade to back up and restore node data. Using snapshots incurs fees (for more information, see Snapshot billing), and the creation progress changes dynamically. After the upgrade, if you no longer need the snapshots, promptly delete them.

    Note

    If you select Upgrade by Replacing System Disk as the upgrade method, we recommend that you enable Auto Snapshot. Creating snapshots incurs resource fees. For more information about the billing, see Snapshot billing.

  4. After you complete the configuration, click Precheck. After the pre-upgrade check passes, follow the on-screen instructions to start the upgrade.

    Note

    If the pre-upgrade check fails or warnings are reported, see Solutions for failed checks or follow the on-screen instructions to view the Check Report and troubleshoot the issues.

    During the upgrade, you can perform the following operations based on the on-screen instructions.

    • Pause: The paused state of the cluster is an intermediate state of the node pool upgrade. We recommend that you do not perform operations on the cluster during this period and complete the upgrade as soon as possible. If a cluster remains in this intermediate state for 7 days, the upgrade process is closed and all upgrade-related events and logs are cleared.

      After the upgrade is paused, the kubelet and container runtime versions of the nodes that have been upgraded cannot be rolled back.

    • Cancel: Cancels the upgrade. After you click Cancel, the kubelet and container runtime versions of the nodes that have been upgraded cannot be rolled back.

    After the upgrade, on the Nodes page, click the name of the target node. On the Basic Information tab, check whether the kubelet version and the container runtime version of the node match the target versions you selected.

Reference information: in-place upgrade and disk replacement upgrade

In-place upgrade and disk replacement upgrade processes

The processes of in-place upgrade and disk replacement upgrade are as follows. When a node pool is upgraded, the system first runs a pre-upgrade check on the nodes. Nodes that fail the pre-upgrade check are skipped in this upgrade. Nodes that pass the pre-upgrade check are upgraded in batches based on the configured maximum number of nodes per batch. In each batch, the number of nodes upgraded at the same time does not exceed the maximum number of nodes per batch. After the current batch is complete, the next batch starts.

For example, if the maximum number of nodes per batch is set to 2, at most 2 nodes are upgraded at the same time in each batch. After the previous batch is complete, the system proceeds to the next batch. If the maximum number of nodes per batch is set to 100, at most 100 nodes are upgraded at the same time in each batch. Valid values of the maximum number of nodes per batch: 1 to 100.

The following figure uses maximum number of nodes per batch = 1, 2, and N as examples to describe the batched upgrade process. The number of nodes upgraded in each batch is 1, 2, and N in sequence.

image

Single-node upgrade logic: in-place upgrade

  1. ACK runs the pre-upgrade check. If a container has a critical exception, such as failing to serve ttrpc requests or processes inside the container failing to respond to signals, the upgrade stops.

  2. ACK saves the current container and Pod states to a temporary directory.

  3. ACK upgrades containerd, crictl, and the related configuration files to the new versions, and then restarts containerd. This does not affect running containers. If you previously modified the /etc/containerd/config.toml configuration on the node, this upgrade overwrites your changes.

  4. ACK verifies that kubelet is running properly and the node is ready.

Single-node upgrade logic: disk replacement upgrade

  1. ACK drains the node. If the node is Schedulable, the system sets it to Unschedulable.

  2. ACK shuts down the Elastic Compute Service (ECS) instance, which stops the node.

  3. ACK replaces the system disk. The system disk ID changes, but the cloud disk category, the instance IP address, and the elastic network interface (ENI) MAC address remain unchanged.

  4. ACK reinitializes the node.

  5. ACK restarts the node and waits for the node to become ready (the node is also set to Schedulable).

    If a node was already set to Unschedulable before the drain, the node does not automatically become Schedulable after the disk replacement upgrade is complete.

FAQ

Is version rollback supported after a node pool upgrade?

After a version upgrade, neither kubelet nor the container runtime supports version rollback. The operating system supports version rollback after an upgrade. When you roll back, make sure that the original image is still supported by the node pool.

Are my services affected during the upgrade?

In-place upgrade: Pods are not restarted, so your services are not affected.

Disk replacement upgrade: the nodes are drained during a disk replacement upgrade. If the Pods implement graceful shutdown logic (Graceful shutdown and zero downtime deployments in Kubernetes) and multiple replicas are deployed across multiple nodes, your services are not affected. To prevent multiple replicas of the same application from being upgraded in the same batch, you can manually set the maximum number of nodes per batch to a value smaller than the number of Pod replicas.

How long does each upgrade batch take?

In-place upgrade: within 5 minutes

Disk replacement upgrade: typically within 8 minutes if no snapshot is involved. If you select the snapshot option for the upgrade, the upgrade is performed after the snapshot is complete, and the duration depends on the time required to create the snapshot. A node pool upgrade tolerates a snapshot for up to 40 minutes. If, within 40 minutes, the snapshot is still not complete, the node upgrade is deemed to have timed out and failed. At this point, the upgrade has not yet started on the failed node. If no business data is stored on the system disk, you can leave the snapshot option unselected to avoid an excessively long upgrade.

Is node data lost during a node upgrade?

When a node runtime upgrade is performed by replacing the node system disk, either keep important data off the system disk, or back it up before the upgrade. Data disks are not affected during the upgrade.

Does the IP address of a node change after the system disk is replaced?

With system disk replacement, the system disk ID changes, but the cloud disk category, the instance IP address, and the ENI MAC address remain unchanged. For more information, see Replace system disk (OS).

How do I upgrade cluster nodes that do not belong to any node pool?

Older clusters created before the node pool feature was released may contain unmanaged nodes, which do not belong to any node pool. You can migrate the unmanaged nodes to a node pool and then upgrade the node pool. For more information, see Migrate unmanaged nodes to a node pool.

What do I do if the Docker directory is not cleaned up and occupies disk space after the node runtime is switched from Docker to containerd?

In addition to the files that the Kubernetes cluster can manage, such as containers, images, and logs, the Docker directory also contains file paths that you created. If you no longer need the Docker directory, delete it manually after the runtime is switched.

Warning

Deleting the Docker directory permanently removes the Docker images, containers, and file paths that the directory contains. The deleted data cannot be restored. Back up the required data before you delete the directory.

How do I restore data from a snapshot?

During a node pool upgrade, you can create snapshots for nodes. A snapshot is retained for 7 days by default. You can also manually delete snapshots in advance. In extreme cases, for example, if you find that data is lost after the upgrade, you can restore the data in the following ways.

  • For an in-place upgrade, for example, when only the kubelet version is upgraded, you can restore data by rolling back the snapshot directly. For more information, see Roll back disk using snapshot.

  • For a disk replacement upgrade, for example, when the operating system or the container runtime is upgraded, you can restore data by creating a cloud disk from the snapshot. For more information, see Create a data disk from a snapshot.

How do I resolve the negative dentry issue?

When the kubelet and containerd upgrade is performed, systemctl daemon-reload is triggered. systemd sets inotify watches on the directories associated with a .path unit and their parent directories (including /, /run, and /run/systemd by default). If a large number of inodes exist under these directories, a kernel soft lockup may occur, which affects node operation.

Because inode counts in these directories cannot be obtained directly, clear dentries during off-peak hours: 

echo 2 > /proc/sys/vm/drop_caches
If you have confirmed that no large number of inodes exist in the directories associated with a .path unit or their parent directories, you can skip this check by selecting Ignore Warnings.

References

Previous：Node pool lifecycleNext：Change the operating system