Accelerate ACK node scale-out with data disk snapshots

How it works

In a default scale-out process, the following network-dependent stages consume most of the time from node creation until the application runs successfully on the node:

• System image pull: The container runtime (containerd) needs to pull images required for system Pods, such as Terway and kube-proxy.

• Application image pull: After a node becomes ready, Pods are scheduled to it, and the container runtime must pull the container images for the application.

When a node pool uses a data disk snapshot to scale out, local load operations replace the network pull operations:

• Faster node readiness: The initialization script checks local paths first. Because the snapshot has system images pre-loaded, the node skips the download and pull processes, reaching the Ready state faster.

• Faster application startup: Because application images are pre-cached in the snapshot, when a Pod is scheduled to a new node, the container runtime can find the required images locally, enabling a rapid application startup.

Step 1: Prepare a node for snapshots

Create an ECS instance with the required ACK core components and application images. You will then create a snapshot from this instance's data disk.

1. Prepare the ECS instance for snapshot creation.

   1. Create a node pool in the cluster with the following key configurations. During this process, ACK automatically saves add-on images such as Terway and the kube-proxy component image to the data disk.

      • Logon Type: Configure a key pair or password to log on to the instance.

      • Expected Number of Nodes: Set to 1.

      • Data Disk: Add at least one data disk.

        If you have been approved for the Initialization Settings feature through the allowlist, select Format and use as a directory of the container runtime. This ensures the container runtime environment is provisioned on the data disk.

   2. After the node becomes ready, remove it from the cluster. When removing the node, do not select Release ECS Instance to retain the underlying ECS instance.

2. Log on to the retained ECS instance and manually add the required components and image files to the data disk.

   1. Log on to the ECS instance and run the lsblk -f command to view the data disk mount path, such as /var/lib/container.

   2. Cache the kubelet component.

      1. Navigate to the data disk mount directory and create a cache directory for ACK components.

         cd /var/lib/container
         mkdir -p ack

      2. Set the environment variables for your region and cluster version.

         # This example uses the cn-shanghai region and version 1.34.1-aliyun.1. Replace them with your actual values.
         export REGION="cn-shanghai"
         export KUBE_VERSION="1.34.1-aliyun.1"

      3. Download, extract, and move the kubelet binary file to the cache directory.

         wget http://aliacs-k8s-$REGION.oss-$REGION-internal.aliyuncs.com/public/pkg/kubernetes/kubernetes-$KUBE_VERSION-linux-amd64.tar.gz
         tar -xvf kubernetes-$KUBE_VERSION-linux-amd64.tar.gz
         mv pkg/kubernetes /var/lib/container/ack/

      4. Clean up the temporary files to keep the snapshot lean.

         rm kubernetes-$KUBE_VERSION-linux-amd64.tar.gz
         rm -rf pkg

3. (Optional) Cache application images.

   Pull frequently used application images to the local instance. These images will be included in the snapshot, accelerating the startup of your application Pods.

   # You can also use 'docker pull' if Docker is available in your environment.
   # The following command uses an Nginx image as an example.
   crictl pull anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6

Step 2: Create a data disk snapshot

Create a snapshot from the data disk with the cached components and images. This snapshot can be reused for future node pool scale-out.

1. Go to ECS console - Instances, and set the target instance to the Stop status.

   For more information about how to stop an instance, see Stop an instance.

2. After the instance is stopped, go to its details page and click the EBS tab.

3. Find the data disk that serves as the container runtime directory. In the Actions column, click Create Snapshot and follow the on-screen instructions to configure the snapshot.

   For more information about how to configure a snapshot, see Manually create a snapshot.

Step 3: Scale out with the snapshot and verify the results

1. Create a node pool and apply the snapshot you created earlier. The key configurations are as follows:

   • Operating System: Select an operating system as needed.

     This feature is not currently supported for ContainerOS 3.5 and later versions.

   • Expected Number of Nodes: Set to 1 or more.

   • Data Disk:

     • Select From Data Disk Snapshot and choose the snapshot that you created.

     • If you have been approved for the Initialization Settings feature through the allowlist, select Format and use as a directory of the container runtime. This formatting process preserves the data in the snapshot.

   • User Data For operating systems other than ContainerOS, add the following script to skip yum updates and speed up node initialization.

     touch /var/.skip-yum

2. After the new nodes are ready, log on to a node and check whether the cache has taken effect successfully.

   1. Check if the image cache is effective.

      Check the kubelet logs. The cache is effective if the logs show no pull records for the pre-cached images.

      # If this command returns no output, or if the output does not contain pull records for pre-cached images like Terway or kube-proxy, the cache is effective.
      journalctl -u kubelet | grep "pulled image"

   2. Check if the kubelet cache is effective.

      Check the ack-deploy log. The absence of the check cached kubernetes failed message indicates that the cache is effective.

      # If this command returns no output, the cached kubelet was used successfully.
      cat /var/log/ack-deploy.log |grep "check cached kubernetes failed"

Billing

The temporary ECS instance used for snapshot creation, its attached cloud disks, and the resulting snapshot all incur charges.

After you complete the process, promptly release the temporary ECS instance. Regularly manage the snapshot lifecycle and delete unneeded snapshots to reduce costs.

FAQ

Related documents

Node initialization process