Scheduling FAQ

How do I prevent Pod startup failures due to insufficient IP addresses in a virtual switch?

The native Kubernetes scheduler cannot see available IP addresses on a Node. Pods scheduled to Nodes with exhausted IP resources fail at startup, often producing many abnormal Pods.

ACK Scheduler resolves this with IP-aware scheduling. It reads the k8s.aliyun.com/max-available-ip annotation on each Node to cap the number of Pods requiring an independent IP. When a Node runs out of IP resources, ACK Scheduler sets a SufficientIP condition on the Node's status, blocking new IP-dependent Pods from being scheduled there.

The kube-scheduler add-on enables this feature automatically. Your cluster must meet these requirements:

• The cluster is an ACK managed cluster Pro Edition with Terway v1.5.7 or later. See Create an ACK managed cluster.

• The kube-scheduler version meets the following requirements:

What is the default scheduling policy of ACK Scheduler?

ACK Scheduler follows the same default policy as the community Kubernetes scheduler. Scheduling involves two steps:

• Filter: Eliminates Nodes where the Pod cannot run. If no Nodes pass filtering, the Pod remains unschedulable.

• Score: Ranks the remaining Nodes and selects the best fit for the Pod.

All Filter and Score plug-ins enabled in the latest ACK Scheduler are listed in kube-scheduler.

How do I avoid scheduling Pods to Nodes with high resource utilization?

The native Kubernetes scheduler uses resource requests, not actual utilization. This can overload some Nodes while others sit idle. ACK provides three approaches:

• Set accurate resource requests and limits. Use resource profiling for container specification recommendations based on historical usage.

• Enable load-aware scheduling. ACK Scheduler factors in actual Node load, analyzes historical data, estimates incoming Pod requirements, and places Pods on lower-utilization Nodes. See Load-aware scheduling.

• Enable load-aware hotspot descheduling. As traffic and workloads change, some Nodes become overloaded. ACK's descheduler detects hot spots and evicts Pods to restore balance. See Using load-aware hotspot descheduling.

A new Node is added to the cluster. Why aren't Pods scheduled to it?

Check the following in order:

1. Node status: The scheduler skips NotReady Nodes. Wait for the Node to become Ready.

2. Scheduling constraints on the Pod: Check for nodeSelector, nodeAffinity, or podAffinity rules on the Pod, or taints on the Node that the Pod does not tolerate.

3. Resource request imbalance: The scheduler uses resource requests, not actual utilization. If existing Nodes have sufficient free requested capacity, the new Node may remain unused. See How do I avoid scheduling Pods to Nodes with high resource utilization?.

Why does scheduling fail due to insufficient CPU or memory when overall cluster utilization is not high?

The scheduler evaluates requested resources, not actual consumption. A Node with low CPU usage can appear "full" if its Pods have high resource requests. Scheduling fails when no single Node has enough requested capacity for the new Pod, even with low cluster-wide utilization.

See How do I avoid scheduling Pods to Nodes with high resource utilization?.

What do I need to know before using descheduling in ACK? Does it restart Pods?

ACK provides descheduling through Koordinator Descheduler. Two things to keep in mind:

• Descheduling only evicts Pods. Koordinator Descheduler evicts running Pods but does not recreate them. The workload controller (Deployment, StatefulSet, and so on) handles recreation, and the scheduler places the new Pod.

• Maintain enough replicas. The old Pod is evicted before the new one starts. Ensure your workload has enough replicas to remain available during eviction.

See Descheduling.

How do I schedule an application to a specific Node?

Add a label to the target Node and a matching nodeSelector to your application's Pod spec. See Schedule an application to a specific node.

In a Deployment, how can I schedule a specific number of Pods to ECS and a specific number to ECI?

Use UnitedDeployment to define separate subsets for ECS and ECI. For example, set replicas: 10 in the ECS subset and replicas: 10 in the ECI subset. See Scale workloads based on UnitedDeployment.

How can I ensure high availability for a workload's Pods during scheduling?

Use podAntiAffinity to spread Pods across zones or Nodes. For example, the following configuration schedules Pods with the security=S2 label to different zones. If the constraint cannot be satisfied, the scheduler falls back to other Nodes.

spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: security
              operator: In
              values:
              - S2
          topologyKey: topology.kubernetes.io/zone

The Kubernetes documentation also covers Pod affinity and anti-affinity and Pod topology spread constraints.

How do I migrate ack-descheduler to Koordinator Descheduler?

The ack-descheduler add-on is deprecated. Migrate to Koordinator Descheduler using the same steps as for Kubernetes Descheduler. See Migrate Kubernetes Descheduler to Koordinator Descheduler and \[Notice\] ack-descheduler migration.

The ack-descheduler add-on in the ACK marketplace is based on open source Kubernetes Descheduler. Versions 0.20.0 and 0.27.1 match their corresponding open source versions.

Why aren't all Pods in a batch scheduled to the Node with the lowest load?

Placing all new Pods on the lowest-load Node would create a hot spot. The load-aware scheduling plug-in adjusts a Node's score when it has newly scheduled Pods whose utilization is not yet reported, preventing over-concentration and balancing load across Nodes.

Besides Node load, what other factors affect scheduling results?

The Kubernetes scheduler uses multiple plug-ins—affinity rules, topology spread constraints, and others—that all contribute to the final Node ranking. Adjust plug-in scoring weights based on your requirements.

After upgrading the scheduler, is the load-aware scheduling feature using the old protocol still supported?

The old protocol requires the alibabacloud.com/loadAwareScheduleEnabled: "true" annotation on Pods. ACK Scheduler is backward-compatible, so upgrading does not break existing Pods. After upgrading, switch to the global load-aware scheduling policy to avoid per-Pod annotation management.

The following tables summarize protocol support by cluster version.

After enabling the CPU Burst configuration, why are my Pods still throttled?

Several conditions can cause CPU throttling to persist:

• Incorrect configuration format. A misconfigured CPU Burst policy has no effect. Verify the format. See Advanced parameter configuration.

• CPU utilization at the cap. If actual CPU usage hits the cfsQuotaBurstPercent limit, throttling persists due to insufficient physical CPU resources. Adjust request and limit values to match actual needs.

• Latency in `cpu.cfs_quota_us` adjustment. CPU Burst modifies cpu.cfs_quota_us and cpu.cfs_burst_us. The cpu.cfs_quota_us value updates only after ack-koordinator detects throttling, introducing a delay. The cpu.cfs_burst_us value is set immediately. For best results, use this feature with Alibaba Cloud Linux.

• Node safety threshold triggered. CPU Burst resets cpu.cfs_quota_us to baseline if Node utilization exceeds the sharePoolThresholdPercent threshold. Adjust the threshold based on your requirements.

Is Alibaba Cloud Linux required for the CPU Burst policy?

No. CPU Burst works on all Alibaba Cloud Linux and CentOS kernels. Alibaba Cloud Linux is recommended because ack-koordinator leverages kernel-level features for finer-grained CPU elasticity. See Enable the CPU Burst feature on the cgroup v1 interface.

After an application uses Batch resources, why does its memory usage suddenly increase?

For containers with a Batch memory limit (kubernetes.io/batch-memory), ack-koordinator sets the cgroup memory limit after container startup. If the application reads the cgroup limit at startup—before ack-koordinator applies it—it may allocate more memory than intended. The OS does not reclaim that memory immediately, so the limit is not enforced until usage drops below it.

To resolve this, tune the application to stay below the Batch limit, or ensure the memory limit parameter is set before the application initializes.

View the current memory limit (in bytes) inside the container:

# The unit is bytes.
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
# Expected output.
1048576000

After increasing the number of CPU cores, why does performance decrease and CPU throttling increase?

Will the dynamic resource overcommitment feature of the legacy ack-slo-manager protocol be supported after upgrading to ack-koordinator?

Yes. ack-koordinator is compatible. The old protocol uses two components in the Pod spec:

• The alibabacloud.com/qosClass annotation

• The alibabacloud.com/reclaimed resource in requests and limits

ack-koordinator recognizes both protocols and calculates resource requests and availability uniformly. Upgrade the add-on without updating existing Pod configurations immediately.

Support for the old protocol ends on July 30, 2023. Update the resource parameters to the latest version as soon as possible.

The following table shows protocol support by scheduler and ack-koordinator version:

Will the CPU Burst feature of the legacy ack-slo-manager protocol be supported after upgrading to ack-koordinator?

Yes. The old protocol uses the alibabacloud.com/cpuBurst annotation. ack-koordinator is fully compatible and handles the upgrade seamlessly.

Support for the old protocol ends on July 30, 2023. Update to the current protocol as soon as possible.

Will the CPU QoS feature of the legacy ack-slo-manager protocol be supported after upgrading to ack-koordinator?

Yes. The old protocol (version 0.8.0 and earlier) enables CPU QoS via the alibabacloud.com/qosClass annotation. ack-koordinator remains compatible and supports gradual migration to the koordinator.sh protocol.

Backward compatibility ends on July 30, 2023. Migrate your Pods to the new protocol promptly.

Will the container memory QoS feature be supported after upgrading from the legacy ack-slo-manager protocol to ack-koordinator?

Yes. The old protocol (version 0.8.0 and earlier) uses the alibabacloud.com/qosClass and alibabacloud.com/memoryQOS annotations. ack-koordinator is backward-compatible with both.

Backward compatibility ends on July 30, 2023. Migrate to the current protocol as soon as possible.

Node utilization has reached the threshold, but Pods are not being evicted. What should I do?

Why does the descheduler restart frequently?

This usually means the descheduler's ConfigMap is missing or misconfigured. Fix the ConfigMap and restart the descheduler. See Advanced configuration parameters.

How do I use load-aware scheduling and hotspot descheduling together?

Enable both features and align their thresholds. When a Node's load exceeds highThresholds, Pods are evicted. Set loadAwareThreshold to the same value as highThresholds to prevent evicted Pods from being rescheduled to the same overloaded Node—especially important in clusters with few Nodes at similar utilization.

See Use load-aware scheduling and Policy description.

What utilization algorithm does the descheduler use?

The descheduler averages utilization over a rolling window and triggers only when the average exceeds the threshold for a sustained period (default: ~10 minutes). Memory calculations exclude page cache, which the OS can reclaim. By contrast, kubectl top node includes page cache. To view the descheduler's memory baseline, use Alibaba Cloud Prometheus.

During a stress test with wrk, the result shows "Socket errors: connect 54,". What should I do?

The wrk client ran out of TCP connections. Enable TCP connection reuse on the stress testing machine.

1. Check whether TCP connection reuse is enabled:

   sudo sysctl -n net.ipv4.tcp_tw_reuse

   An output of 0 or 2 indicates TCP reuse is not fully enabled.

2. Enable TCP connection reuse:

   sudo sysctl -w net.ipv4.tcp_tw_reuse=1

3. Rerun the wrk test. The Socket errors: connect 54, ... message should no longer appear.

These commands apply only to the stress testing machine. After the test, restore the original setting with sysctl -w net.ipv4.tcp_tw_reuse=<original_value>.

Why is no data displayed in the cluster colocation benefits section on the k8s-reclaimed-resource tab?

1. Verify that ack-koordinator is installed:

   1. Log on to the ACK console. In the left navigation pane, click Clusters.

   2. Click the cluster name. In the left navigation pane, choose Applications > Helm.

   3. On the Helm page, check whether ack-koordinator is listed. If not, install it. See Install and manage add-ons.

2. If the colocation dashboard shows no data, verify that the kube_node_labels metric is being collected:

   1. Log on to the ARMS console.

   2. In the left navigation pane, choose Managed Service for Prometheus > Instances.

   3. Select the region, click the Prometheus instance name, and click Metric Management.

   4. Click Metrics, search for kube_node_labels, and verify that the metric has data.

Can I use Arm-based preemptible instances?

Yes. See Use spot instances.

What are the limitations of using Arm-based Nodes in an ACK cluster?

On the Add-ons page, only the following add-on categories support the Arm architecture:

• Core components

• Logging and monitoring

• Storage

• Network

Add-ons from the marketplace do not support the Arm architecture.