Manually Rotate and Update ACK Dedicated Cluster Master Nodes-Container Service for Kubernetes(ACK)-阿里云帮助中心

Important

ACK dedicated clusters are no longer available for creation as of August 21, 2024. For production workloads, use ACK managed Pro clusters, which offer improved reliability, security, and scheduling efficiency. To create a new cluster, see Create an ACK managed Pro cluster. To migrate an existing cluster, see Migrate an ACK dedicated cluster to an ACK managed Pro cluster.

When migrating an ACK dedicated cluster to an ACK managed Pro cluster is not feasible, manually rotate the master nodes to change their OS or zone while keeping the cluster in service. Follow this guide to manually rotate master nodes and update their configuration if any of the following applies:

Your master nodes run an OS that has reached end-of-maintenance (such as CentOS or Alibaba Cloud Linux 2), and you want to upgrade to container-optimized Alibaba Cloud Linux 3.
Your master nodes are concentrated in a small number of zones, and you want to distribute them across additional zones.
You want to move your master nodes to different zones.

Use cases covered in this guide

Use case	What changes	Section
Change the OS	Replaces the OS image on master nodes (for example, with the Alibaba Cloud Linux 3 container-optimized image)	Use case 1: Change the master node OS
Change the zone	Moves master nodes to different zones or distributes them across more zones	Use case 2: Change the master node zone

Prerequisites

Before you begin, verify the following:

Your cluster runs Kubernetes 1.20 or later. Depending on your cluster version, consider upgrading to a newer version or migrating to an ACK managed Pro cluster. For more information, see Product Change: Announcement on Discontinuation of Support for Upgrades of Some Low-Version ACK Dedicated Clusters.
Master node rotation uses ECS Cloud Assistant to access nodes. Verify that Cloud Assistant is in healthy status on all current master nodes. To check the status and resolve issues, see Viewing Cloud Assistant status and handling abnormal status.

All etcd instances are healthy. Log on to a master node and run the following command to verify. Replace local_ip with the IP address of the master node instance.

ETCDCTL_API=3 etcdctl --cacert=/var/lib/etcd/cert/ca.pem --cert=/var/lib/etcd/cert/etcd-server.pem --key=/var/lib/etcd/cert/etcd-server-key.pem --endpoints=https://${local_ip}:2379 endpoint --cluster health

The ConfigMap ack-admin-config exists in the kube-system namespace. If it does not exist, create it by running the following commands:

wget https://aliacs-k8s-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/public/pkg/ackadmin/ackadmin-1.0-linux-amd64.tar.gz
tar -xvf ackadmin-1.0-linux-amd64.tar.gz
cp pkg/ackadmin/1.0/ackadmin /usr/local/bin/ackadmin

/usr/local/bin/ackadmin config-migrate-from-current-pod --clusterId=${CLUSTER_ID} -v 3 # Replace ${CLUSTER_ID} with your actual cluster ID.

Usage notes

Back up cluster applications using the backup center before starting the rotation.
The SSH maintenance port exposed through the API server Classic Load Balancer (CLB) instance during cluster creation will be unbound when the master nodes are released. To continue using the SSH port for cluster maintenance, log on to the CLB console and manually attach the backend virtual server group for the API server CLB SSH port.
Perform the rotation during off-peak hours. Avoid making other changes to the cluster—such as upgrades or application deployments—during the rotation to avoid unexpected results.

How it works

Master node rotation follows a scale-out then scale-in approach:

Update the control plane configuration so that newly added master nodes use the new settings.
Add one new master node with the updated configuration.
Verify that the new node is healthy.
Remove one old master node.
Repeat steps 2–4 until all master nodes are replaced, then restore the original node count.

General rotation procedure

The following steps apply to all use cases. See the Use cases sections for specific parameter values.

Step 1: Query the current configuration

Call DescribeClusterDetail to retrieve the current control plane configuration (control_plane_config) and vSwitch configuration (vswitch_ids) as the basis for subsequent modifications.

Key parameters:

For complete parameter descriptions, see DescribeClusterDetail.

Parameter	Description
`vswitch_ids`	The vSwitches used by the cluster control plane. These determine the zones where master nodes reside. To place new master nodes in a specific zone, replace this with the vSwitch IDs for that zone.
`image_type`	The OS image type for control plane nodes. Select a non-end-of-maintenance supported OS type, and review the related limits.
`runtime`	The container runtime. Kubernetes 1.24 and later no longer support Docker as a built-in container runtime—use containerd instead.
`instance_types`	The instance type for master nodes. See ECS instance type configuration recommendations for guidance.
`size`	The number of master nodes. Typically 3 or 5.

Example output

{
    "vswitch_ids": [
        "vsw-bp1c1j6wbqehaiqmz*****"
    ],
    "control_plane_config": {
        "image_type": "AliyunLinux3",
        "node_port_range": "30000-32767",
        "key_pair": "ack",
        "charge_type": "PostPaid",
        "runtime": "containerd",
        "deploymentset_id": "ds-bp10b35imuam5am******",
        "cloud_monitor_flags": true,
        "instance_types": [
            "ecs.c6.xlarge"
        ],
        "system_disk_category": "cloud_essd",
        "system_disk_size": 50,
        "system_disk_performance_level": "PL1",
        "security_hardening_os": true,
        "size": 3
    }
}

Step 2: Update the target configuration

Call ModifyCluster to update the relevant control plane settings for your scenario.

Important

Submit a complete set of parameter values—not just the fields you are changing.

Step 3: Verify the configuration change

Call DescribeClusterDetail again and confirm that control_plane_config matches your expected values. Newly scaled-out master nodes will use this updated configuration.

Step 4: Scale out one new master node

Call ModifyCluster and increase control_plane_config.size by 1. For example, increase it from 3 to 4.

{
    "control_plane_config": {
        "size": 4
    }
}

Step 5: Verify the new node

Log on to the ACK console. On the Clusters page, click the target cluster name, then choose Nodes > Nodes in the left navigation pane. Wait for the new master node to be created, then confirm that:

The node configuration is correct.
The node status is healthy.
The cluster is operating normally.

Step 6: Remove one old master node

After confirming cluster stability, call DeleteClusterNodes and specify an old master node for removal. For details, see DeleteClusterNodes API reference.

Step 7: Repeat until all nodes are replaced

Repeat steps 3–6 until all master nodes are updated. When the rotation is complete, restore the master node count to its original number (typically 3 or 5).

Use cases

Use case 1: Change the master node OS

This scenario uses the container-optimized Alibaba Cloud Linux 3 image, which requires Kubernetes 1.26 or later.

Call DescribeClusterDetail to retrieve the current control_plane_config and vswitch_ids.

Call ModifyCluster to change the OS image type.

Important

Submit a complete set of parameter values.

{
    "vswitch_ids": [
        "vsw-bp1c1j6wbqehaiqmz*****"
    ],
    "control_plane_config": {
        "image_type": "AliyunLinux3ContainerOptimized",
        "node_port_range": "30000-32767",
        "key_pair": "ack",
        "charge_type": "PostPaid",
        "runtime": "containerd",
        "deploymentset_id": "ds-bp10b35imuam5am******",
        "cloud_monitor_flags": true,
        "instance_types": [
            "ecs.c6.xlarge"
        ],
        "system_disk_category": "cloud_essd",
        "system_disk_size": 50,
        "system_disk_performance_level": "PL1",
        "security_hardening_os": true,
        "size": 3
    }
}

Call DescribeClusterDetail to confirm the change. If the returned control_plane_config shows image_type as AliyunLinux3ContainerOptimized, the update succeeded. Newly scaled-out nodes will run the Alibaba Cloud Linux 3 container-optimized image.
Call ModifyCluster to scale out master nodes by increasing the count by 1.
If you currently have 3 master nodes, increase to 4.
```
{
    "control_plane_config": {
        "size": 4
    }
}
```
Log on to the ACK console. On the Clusters page, click the target cluster name, then choose Nodes > Nodes. Wait for the scale-out to complete. Confirm that the new master node uses the expected OS and that the node and cluster are healthy.
Call DeleteClusterNodes to remove one old master node. For details, see DeleteClusterNodes API reference.
If the cluster remains healthy, repeat steps 3–6 until all master nodes are updated.

Use case 2: Change the master node zone

Call DescribeClusterDetail to retrieve the current control_plane_config and vswitch_ids.

Call ModifyCluster to update the vSwitch configuration. Set vswitch_ids to the vSwitch ID of the target zone.

Important

Submit a complete set of parameter values.

{
    "vswitch_ids": [
        "vsw-bp1c1j6wbqehaiqmz*****"
    ],
    "control_plane_config": {
        "image_type": "AliyunLinux3",
        "node_port_range": "30000-32767",
        "key_pair": "ack",
        "charge_type": "PostPaid",
        "runtime": "containerd",
        "deploymentset_id": "ds-bp10b35imuam5am******",
        "cloud_monitor_flags": true,
        "instance_types": [
            "ecs.c6.xlarge"
        ],
        "system_disk_category": "cloud_essd",
        "system_disk_size": 50,
        "system_disk_performance_level": "PL1",
        "security_hardening_os": true,
        "size": 3
    }
}

Call DescribeClusterDetail to confirm the change. If the returned vswitch_ids matches the value you submitted, the update succeeded. Newly scaled-out nodes will be deployed in the target zone.
Call ModifyCluster to scale out master nodes by increasing the count by 1.
If you currently have 3 master nodes, increase to 4.
```
{
    "control_plane_config": {
        "size": 4
    }
}
```
Log on to the ACK console. On the Clusters page, click the target cluster name, and then choose Nodes > Nodes. Wait for the scale-out to complete. Confirm that the new master node is in the expected zone and that the node and cluster are healthy.
Call DeleteClusterNodes to remove one old master node. For details, see DeleteClusterNodes API reference.
(Optional) To move additional master nodes to new zones, repeat steps 3–6 until all master nodes are distributed across the target zones.