Node auto scaling FAQ

Index

Known limitations

Inability to precisely predict available node resources

The available resources on a new node may not match the instance type's specifications. This is because the underlying OS and system daemons on the ECS instance consume resources. For more information, see Why is the memory size of my ECS instance different from the size defined in its instance type? Due to this overhead, the resource estimates used by the cluster-autoscaler may be slightly higher than the actual allocatable resources on a new node. When you configure pod requests, note the following:

• The total requested resources for a pod, including CPU, memory, and disk, must be less than the instance type's specifications. As a best practice, do not request more than 70% of a node's total capacity.

• To determine if a node has sufficient resources, the cluster-autoscaler considers only the resource requests of Kubernetes pods, including pending pods and DaemonSet pods. If you run static pods that are not managed as DaemonSets, you must manually account for and reserve resources for them.

• If a pod requests a large amount of resources, such as more than 70% of a node's resources, test in advance to confirm that the pod can be scheduled to a node of the same instance type.

Limited support for scheduling policies

The cluster-autoscaler supports only a limited set of scheduling policies for determining if an unschedulable pod can be scheduled to a node pool with auto scaling enabled. For more information, see Supported scheduling policies.

Only resource-based ResourcePolicy is supported

When you use a ResourcePolicy to customize the priority of elastic resources, only policies of the resource type are supported. For more information, see Customize priority-based scheduling for elastic resources.

apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    app: nginx 
  units:
  - resource: ecs
  - resource: eci

Cannot scale out a specific instance type

If your node pool is configured with multiple instance types, you cannot direct the cluster-autoscaler to provision a specific instance type during a scale-out. The cluster-autoscaler models the capacity of the entire node pool based on the minimum value for each resource dimension across all configured instance types. For more information, see Resource calculation in multi-type scaling groups.

Zone-dependent pods cannot trigger scale-out

If your node pool spans multiple availability zones, a pod with a dependency on a specific zone may not trigger a scale-out. This can occur if the pod requires a specific availability zone due to a PersistentVolumeClaim (PVC) bound to a volume in that zone, or a nodeSelector that targets the zone. In these cases, the cluster-autoscaler might fail to add a node in the required availability zone. For more scenarios in which the cluster-autoscaler fails to add nodes, see Reasons for scale-out failures.

cluster-autoscalerSupported scheduling policies

The following scheduling policies are used:

• PodFitsResources

• GeneralPredicates

• PodToleratesNodeTaints

• MaxGCEPDVolumeCount

• NoDiskConflict

• CheckNodeCondition

• CheckNodeDiskPressure

• CheckNodeMemoryPressure

• CheckNodePIDPressure

• CheckVolumeBinding

• MaxAzureDiskVolumeCount

• MaxEBSVolumeCount

• ready

• NoVolumeZoneConflict

cluster-autoscalerSimulated resources for scheduling

The cluster-autoscaler can simulate and evaluate the following resources:

cpu
memory
sigma/eni
ephemeral-storage
aliyun.com/gpu-mem (shared GPUs only)
nvidia.com/gpu

To use other resource types, see Configure custom resources.

Reasons for scale-out failures

This can happen for several reasons. Review the following to diagnose the issue:

• Node auto scaling only works on node pools where it is enabled. Make sure that the node auto scaling feature is enabled and that the scaling mode of the node pool is set to Auto. For more information, see Enable node auto scaling.

• The instance types in the scaling group cannot meet the pod's resource requests. The advertised specifications of an ECS instance type represent its total capacity. However, a portion of these resources is reserved by ACK to ensure the stable operation of the OS kernel, system services, and essential Kubernetes daemons. This reservation creates a critical difference between the node's total capacity and its allocatable resources—the amount of CPU, memory, and storage that is actually available for pods.

  The standard cluster-autoscaler bases its scaling decisions on the resource reservation policy from Kubernetes v1.28 and earlier.

  To apply the resource reservation policy for versions 1.28 and later, we recommend switching to Enable node instant scaling (versions 1.28 and later include a new built-in resource reservation algorithm), or manually configuring and maintaining custom resources for a node pool (by defining and maintaining resource reservation values in the node pool).

  • During instance creation, some resources are consumed by components such as virtualization and the operating system. For more information, see Why is the memory size of my instance different from the size defined by its instance type?.

  • Running components such as kubelet, kube-proxy, Terway, and Container Runtime requires some node resources. For more information, see Node resource reservation policy.

  • System components are installed on nodes by default. A pod's resource requests must be less than the resources defined by the instance type.

• A pod with constraints on availability zones cannot trigger the scale-out of a node pool configured with multiple availability zones.

• The authorization steps were not completed. Authorization is a cluster-level operation and must be performed for each cluster. For more information about authorization, see the Usage notes section.

• The node pool with auto scaling enabled experiences exceptions. The autoscaler includes a dampening mechanism to prevent repeated failures. If it provisions nodes that subsequently fail to join the cluster or remain in a NotReady state for an extended period, it temporarily pauses further scaling operations. Common exceptions include:

  • The instance fails to join the cluster and times out.

  • The node is NotReady and times out.

  This dampening approach prevents further scaling until the faulty nodes are resolved, which ensures future accuracy.

• The cluster-autoscaler itself runs as a pod within the cluster. If there are no worker nodes, the autoscaler pod cannot run and therefore cannot provision new nodes. Configure your node pools with a minimum of two nodes to ensure high availability for core cluster components.

  If your use case requires scaling out from zero nodes or scaling down to zero nodes, use the node instant scaling feature.

Resource calculation in multi-type scaling groups

For a scaling group with multiple instance types, the cluster-autoscaler models the group's capacity based on the minimum value for each resource (like CPU and memory) across all configured instance types.

For example, consider a scaling group configured with two instance types: one with 4 vCPU and 32 GiB of memory, and another with 8 vCPU and 16 GiB of memory. To determine the baseline for this group, the autoscaler calculates the minimums for each resource: minimum CPU is min(4, 8) = 4 vCPU, and minimum memory is min(32, 16) = 16 GiB. As a result, the autoscaler treats this entire scaling group as if it can only provision nodes with 4 vCPU and 16 GiB of memory. Therefore, if a pending pod's requests exceed 4 vCPU or 16 GiB of memory, it will not trigger a scale-out.

If you have configured multiple instance types and also need to account for resource reservation, see Reasons for scale-out failures.

Node pool selection for scale-out

When a pod is unschedulable, the cluster-autoscaler simulates which auto-scaled node pool could accommodate it. This simulation evaluates each node pool based on its configuration, including labels, taints, and available instance types. A node pool is considered a candidate for scale-out if the simulation shows that a new node from that pool would allow the pod to be scheduled. If multiple node pools meet this condition, the node auto scaling component defaults to the least-waste principle. This strategy selects the node pool that leaves the fewest unused resources after scheduling the pod.

Configure custom resources

You can configure ECS tags with a specific prefix for a node pool with scaling enabled. This allows the cluster-autoscaler to identify the custom resources available in the node pool or the precise values of specified resources.

k8s.io/cluster-autoscaler/node-template/resource/{resource_name}:{resource_size}

Example:

k8s.io/cluster-autoscaler/node-template/resource/hugepages-1Gi:2Gi

Reasons for auto scaling enablement failures

Enabling auto scaling for a node pool may fail for the following reasons:

• The node pool is the default node pool. The node auto scaling feature cannot be enabled on the default node pool.

• You cannot enable auto scaling on a node pool that contains manually added nodes. To resolve this, remove the manually added nodes first or create a new node pool with auto scaling enabled.

• The node pool contains subscription-based instances. The node auto scaling feature does not support nodes with a subscription billing method.

Why does cluster-autoscaler fail to scale down nodes?

Several reasons can prevent the cluster-autoscaler from scaling in a node. Check for the following cases:

• The cluster-autoscaler does not consider a node for scale-in if the total resource requests of the pods running on it exceed the configured scale-in utilization threshold.

• By default, the cluster-autoscaler does not remove nodes that are running pods from the kube-system namespace.

• The node cannot be scaled in if it is running a pod with a strict scheduling constraint (such as a nodeSelector or a strong nodeAffinity) that prevents it from being rescheduled onto any other available node in the cluster.

• The pods on the node have a PodDisruptionBudget, and its minimum has been reached.

For more frequently asked questions and answers from the open source community, see the cluster-autoscaler FAQ.

Control DaemonSet eviction

The cluster-autoscaler decides whether to evict DaemonSet pods based on the Evict DaemonSet Pods setting. This is a cluster-wide configuration that applies to all DaemonSet pods in the cluster. For more information, see Step 1: Enable node auto scaling for a cluster. You can override this global setting for individual DaemonSets by adding an annotation to their pods. To explicitly enable eviction for a specific DaemonSet's pods, add the "cluster-autoscaler.kubernetes.io/enable-ds-eviction": "true" annotation.

Similarly, to explicitly disable eviction for a specific DaemonSet's pods, add the annotation "cluster-autoscaler.kubernetes.io/enable-ds-eviction": "false" to the DaemonSet pod.

What types of Pods can prevent the cluster-autoscaler from removing a node?

The cluster-autoscaler cannot remove a node if it contains pods not created by a native Kubernetes controller (such as a Deployment, ReplicaSet, Job, or StatefulSet), or if pods on the node cannot be safely terminated or migrated. For a comprehensive list of conditions that can block a node scale-in, see What types of pods can prevent CA from removing a node? in the Kubernetes documentation.

cluster-autoscalerCRD support

No, the cluster-autoscaler currently supports only standard Kubernetes objects and does not support Kubernetes CRDs.

How to delay cluster-autoscaler's scale-up reaction time for unschedulable Pods?

You can set a scale-out delay for each pod by using the cluster-autoscaler.kubernetes.io/pod-scale-up-delay annotation. The cluster-autoscaler will only consider the pod for a scale-out if it remains unschedulable after this delay period has passed. This gives the native Kubernetes scheduler extra time to try to place the pod on existing nodes before a scale-out is triggered. Example annotation: "cluster-autoscaler.kubernetes.io/pod-scale-up-delay": "600s".

How to use Pod annotations to affect cluster-autoscaler node scale-in?

Use the pod annotation cluster-autoscaler.kubernetes.io/safe-to-evict to tell the cluster-autoscaler whether a pod can be safely evicted during a scale-in.

• To prevent a node from being scaled in, add the "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation to a pod running on the node. The presence of this pod will block the node from being terminated by the autoscaler.

• To allow a node to be scaled in, add the "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" annotation to a pod. This explicitly marks the pod as safe to evict during a scale-in operation.

How to prevent cluster-autoscaler from scaling in a node?

To prevent the cluster-autoscaler from scaling in a specific node, add the "cluster-autoscaler.kubernetes.io/scale-down-disabled": "true" annotation to it. To add this annotation, run the following kubectl command, replacing <nodename> with the name of your target node:

kubectl annotate node <nodename> cluster-autoscaler.kubernetes.io/scale-down-disabled=true

Upgrade the cluster-autoscaler

To upgrade the cluster-autoscaler on a cluster where auto scaling is enabled, follow these steps:

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.

3. To the right of Edit, click Node Scaling. In the panel that appears, click OK to upgrade the component to the latest version.

What operations trigger cluster-autoscaler to update automatically?

To ensure that the cluster-autoscaler configuration is up-to-date and its version remains compatible with your cluster, the following operations trigger an automatic update or reconciliation of the cluster-autoscaler:

• Updating the auto scaling configuration.

• Creating, deleting, or updating a node pool where auto scaling is enabled.

• Successfully upgrading the cluster's Kubernetes version.

An ACK managed cluster has been granted role permissions, but node scaling activities still fail to run properly?

This issue occurs if the required token (addon.aliyuncsmanagedautoscalerrole.token) is missing from a Secret in the kube-system namespace. By default, ACK uses the cluster's WorkerRole to enable auto scaling capabilities, and this token is essential for authenticating those operations. To resolve this, re-apply the required policy to the cluster's WorkerRole by using the authorization wizard in the ACK console.

1. On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.

2. On the Node Pools page, to the right of Node Scaling, click Enable.

3. Follow the on-screen instructions to grant the required permissions to the KubernetesWorkerRole role and attach the AliyunCSManagedAutoScalerRolePolicy system policy. The entry point is shown below.

   

4. Manually restart the cluster-autoscaler Deployment (for node auto scaling) or the ack-goatscaler Deployment (for node instant scaling) in the kube-system namespace to apply the changes immediately.