Use hotspot-aware descheduling to balance node loads

Limitations

• Only ACK managed Pro clusters are supported.

• Required component versions:

  Component	Version
  ACK Scheduler	v1.22.15-ack-4.0 or later, v1.24.6-ack-4.0 or later
  ack-koordinator	v1.1.1-ack.1 or later
  Helm	v3.0 or later

Billing

ack-koordinator is free to install and use. Additional charges may apply in these scenarios:

• Worker node resources: ack-koordinator is a self-managed component that consumes worker node resources. Configure resource requests for each module during installation.

• Prometheus custom metrics: If you enable Enable Prometheus Monitoring for ACK-Koordinator and use Managed Service for Prometheus, the exposed metrics are counted as custom metrics and incur fees. Fees depend on cluster size and number of applications. Review billing of Prometheus instances before enabling this feature. Query usage data to monitor your resource consumption.

How it works

Execution cycle

The Koordinator Descheduler runs periodically. Each execution cycle has three stages:

1. Data collection: Fetches node and workload information and resource utilization data.

2. Plugin execution (LowNodeLoad example):

   1. Identifies hot spot nodes based on highThresholds and lowThresholds.

   2. Traverses all hot spot nodes, scores the eligible Pods, and sorts them for eviction. See Pod scoring policy.

   3. Checks each candidate Pod against migration constraints—cluster capacity, resource utilization, and replica ratio limits. See Load-aware hot spot descheduling policies.

   4. Marks passing Pods as migration candidates and skips the rest.

3. Pod eviction and migration: Evicts migration candidates via the eviction API.

Node classification

The LowNodeLoad plugin classifies nodes into three categories based on two thresholds: lowThresholds (idle threshold) and highThresholds (hot spot threshold).

Example with lowThresholds = 45% and highThresholds = 70%:

1. Idle node: Resource utilization is below lowThresholds (< 45%).

2. Normal node: Resource utilization is between lowThresholds and highThresholds (45%–70%). This is the target range.

3. Hot spot node: Resource utilization exceeds highThresholds (> 70%). Pods are evicted until the node load drops to 70% or below.

Resource utilization data is updated every minute and reflects the 5-minute average.

If all nodes exceed lowThresholds, the cluster load is considered high and descheduling is suspended, even if some nodes exceed highThresholds.

Prerequisites

Before you begin, make sure you have:

• An ACK managed Pro cluster

• ACK Scheduler v1.22.15-ack-4.0 or later, ack-koordinator v1.1.1-ack.1 or later, and Helm v3.0 or later

Step 1: Enable descheduling in ack-koordinator

• New installation: Install ack-koordinator and select Enable Descheduling For Ack-koordinator on the configuration page.

• Existing installation: On the configuration page, select Enable Descheduling For Ack-koordinator. See Modify ack-koordinator.

Step 2: Enable the LowNodeLoad plugin

1. Create a koord-descheduler-config.yaml ConfigMap to enable the LowNodeLoad plugin.

   # koord-descheduler-config.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: koord-descheduler-config
     namespace: kube-system
   data:
     koord-descheduler-config: |
       # System configuration for koord-descheduler. Do not modify this section.
       apiVersion: descheduler/v1alpha2
       kind: DeschedulerConfiguration
       leaderElection:
         resourceLock: leases
         resourceName: koord-descheduler
         resourceNamespace: kube-system
       deschedulingInterval: 120s  # Execution interval. The descheduler runs every 120s.
                                   # Must not exceed detectorCacheTimeout (default: 5m).
       dryRun: false               # Set to true to run in read-only mode (no evictions).
       # End of system configuration.
   
       profiles:
       - name: koord-descheduler
         plugins:
           balance:
             enabled:
               - name: LowNodeLoad      # Enable the LowNodeLoad plugin for hot spot descheduling.
           evict:
             enabled:
               - name: MigrationController  # Enable the eviction and migration controller.
   
         pluginConfig:
         - name: MigrationController
           args:
             apiVersion: descheduler/v1alpha2
             kind: MigrationControllerArgs
             defaultJobMode: EvictDirectly
   
         - name: LowNodeLoad
           args:
             apiVersion: descheduler/v1alpha2
             kind: LowNodeLoadArgs
   
             # A node is idle if usage of ALL resources is below lowThresholds.
             lowThresholds:
               cpu: 20    # 20% CPU utilization
               memory: 30 # 30% memory utilization
   
             # A node is a hot spot if usage of ANY resource exceeds highThresholds.
             highThresholds:
               cpu: 50    # 50% CPU utilization
               memory: 60 # 60% memory utilization
   
             # Scopes descheduling to specific namespaces.
             # include and exclude are mutually exclusive—configure only one.
             evictableNamespaces:
               include:
                 - default
               # exclude:
               #   - "kube-system"
               #   - "koordinator-system"

2. Apply the ConfigMap to the cluster.

   kubectl apply -f koord-descheduler-config.yaml

3. Restart the Koordinator Descheduler to load the new configuration.

   kubectl -n kube-system scale deploy ack-koord-descheduler --replicas 0
   kubectl -n kube-system scale deploy ack-koord-descheduler --replicas 1

Step 3 (optional): Enable load-aware scheduling

For optimal load balancing, enable load-aware scheduling so the ACK Scheduler avoids placing Pods on hot spot nodes after eviction.

Set loadAwareThreshold to match highThresholds. Mismatched values may cause evicted Pods to be rescheduled onto hot spot nodes, especially with few nodes at similar utilization.

Step 4: Verify descheduling

This example uses a three-node cluster where each node has 104 cores and 396 GiB of memory.

1. Create a stress-demo.yaml file with the following content.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: stress-demo
     namespace: default
     labels:
       app: stress-demo
   spec:
     replicas: 6
     selector:
       matchLabels:
         app: stress-demo
     template:
       metadata:
         name: stress-demo
         labels:
           app: stress-demo
       spec:
         containers:
           - args:
               - '--vm'
               - '2'
               - '--vm-bytes'
               - '1600M'
               - '-c'
               - '2'
               - '--vm-hang'
               - '2'
             command:
               - stress
             image: polinux/stress
             imagePullPolicy: Always
             name: stress
             resources:
               limits:
                 cpu: '2'
                 memory: 4Gi
               requests:
                 cpu: '2'
                 memory: 4Gi
         restartPolicy: Always

2. Deploy the stress-testing workload.

   kubectl create -f stress-demo.yaml

   Expected output:

   deployment.apps/stress-demo created

3. Verify the Pods are running and note their assigned node.

   kubectl get pod -o wide

   Expected output:

   NAME                           READY   STATUS    RESTARTS   AGE   IP            NODE                     NOMINATED NODE   READINESS GATES
   stress-demo-588f9646cf-s****   1/1     Running   0          82s   10.XX.XX.53   cn-beijing.10.XX.XX.53   <none>           <none>

4. Increase the load on cn-beijing.10.XX.XX.53 and check node utilization.

   kubectl top node

   Expected output:

   NAME                      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
   cn-beijing.10.XX.XX.215   17611m       17%    24358Mi         6%
   cn-beijing.10.XX.XX.53    63472m       63%    11969Mi         3%

   Node cn-beijing.10.XX.XX.53 is at 63% CPU, exceeding the 50% hot spot threshold. Node cn-beijing.10.XX.XX.215 is at 17% CPU, below the 20% idle threshold.

5. Enable the LowNodeLoad plugin as described in Step 2: Enable the LowNodeLoad plugin.

6. Watch for Pod changes.

   By default, a node must exceed highThresholds for five consecutive checks (~10 minutes at the default 120-second interval) to be classified as a hot spot.

   kubectl get pod -w

   Expected output:

   NAME                           READY   STATUS             RESTARTS   AGE   IP             NODE                     NOMINATED NODE   READINESS GATES
   stress-demo-588f9646cf-s****   1/1     Terminating        0          59s   10.XX.XX.53    cn-beijing.10.XX.XX.53   <none>           <none>
   stress-demo-588f9646cf-7****   1/1     ContainerCreating  0          10s   10.XX.XX.215   cn-beijing.10.XX.XX.215  <none>           <none>

7. Check the eviction event.

   kubectl get event | grep stress-demo-588f9646cf-s****

   Expected output:

   2m14s   Normal   Evicting        podmigrationjob/00fe88bd-****   Pod "default/stress-demo-588f9646cf-s****" evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(68.53%)>threshold(50.00%)"
   101s    Normal   EvictComplete   podmigrationjob/00fe88bd-****   Pod "default/stress-demo-588f9646cf-s****" has been evicted
   2m14s   Normal   Descheduled     pod/stress-demo-588f9646cf-s****   Pod evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(68.53%)>threshold(50.00%)"
   2m14s   Normal   Killing         pod/stress-demo-588f9646cf-s****   Stopping container stress

   The Pod on the hot spot node has been evicted and migrated to cn-beijing.10.XX.XX.215.

Advanced configuration

All Koordinator Descheduler parameters are configured in the ConfigMap from Step 2.

# koord-descheduler-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: koord-descheduler-config
  namespace: kube-system
data:
  koord-descheduler-config: |
    # System configuration. Do not modify this section.
    apiVersion: descheduler/v1alpha2
    kind: DeschedulerConfiguration
    leaderElection:
      resourceLock: leases
      resourceName: koord-descheduler
      resourceNamespace: kube-system
    deschedulingInterval: 120s  # Execution interval. Must not exceed detectorCacheTimeout.
    dryRun: false               # Set to true for read-only mode (no evictions).
    # End of system configuration.

    profiles:
    - name: koord-descheduler
      plugins:
        deschedule:
          disabled:
            - name: "*"  # All plugins disabled by default (shown for reference only).
        balance:
          enabled:
            - name: LowNodeLoad
        evict:
          disabled:
            - name: "*"  # All plugins disabled by default (shown for reference only).
          enabled:
            - name: MigrationController

      pluginConfig:
      - name: MigrationController
        args:
          apiVersion: descheduler/v1alpha2
          kind: MigrationControllerArgs
          defaultJobMode: EvictDirectly
          maxMigratingPerNode: 1       # Max Pods migrating simultaneously per node. 0 = unlimited.
          maxMigratingPerNamespace: 1  # Max Pods migrating simultaneously per namespace.
          maxMigratingPerWorkload: 1   # Max Pods migrating simultaneously per workload (e.g., Deployment).
          maxUnavailablePerWorkload: 2 # Max unavailable replicas per workload during migration.
          evictLocalStoragePods: false # Whether to evict Pods using HostPath or emptyDir volumes.
          objectLimiters:
            workload:               # At most 1 replica migrated per workload within 5 minutes.
              duration: 5m
              maxMigrating: 1

      - name: LowNodeLoad
        args:
          apiVersion: descheduler/v1alpha2
          kind: LowNodeLoadArgs

          lowThresholds:
            cpu: 20
            memory: 30
          highThresholds:
            cpu: 50
            memory: 60

          anomalyCondition:
            consecutiveAbnormalities: 5  # Number of consecutive checks above highThresholds
                                          # before a node is classified as a hot spot.
                                          # Counter resets after eviction.

          detectorCacheTimeout: "5m"  # Cache duration for hot spot checks.
                                      # Must be >= deschedulingInterval.

          evictableNamespaces:
            include:
              - default
            # exclude:
            #   - "kube-system"
            #   - "koordinator-system"

          nodeSelector:  # Process only the specified nodes.
            matchLabels:
              alibabacloud.com/nodepool-id: np77f520e1108f47559e63809713ce****

          podSelectors:  # Process only the specified Pods.
          - name: lsPods
            selector:
              matchLabels:
                koordinator.sh/qosClass: "LS"

FAQ

Next steps

• Use load-aware pod scheduling

• Install and manage the ack-koordinator component