Configure hierarchical quotas with guaranteed minimums to share idle cluster capacity across teams.
Native Kubernetes ResourceQuota enforces fixed resource caps, often leaving resources idle when teams underuse their quota. ACK implements capacity scheduling through the scheduling framework extension, replacing this static model with elastic quota groups: idle resources are shared and reclaimed when owners need them. This improves cluster utilization without compromising resource guarantees.
Prerequisites
Ensure that you have:
-
An ACK managed Pro cluster running Kubernetes 1.20 or later
Key concepts
ElasticQuotaTree is a CustomResourceDefinition (CRD) defining a hierarchy of elastic quota groups. Each node represents a quota boundary. Leaf nodes map to one or more namespaces, and pods in those namespaces are scheduled within their leaf node's quota limits.
The two core fields in each quota node are:
| Field | Meaning |
|---|---|
min |
Guaranteed resources. The scheduler ensures this amount is available, reclaiming borrowed resources if needed. |
max |
Maximum resources the quota node can use, including idle resources borrowed from other nodes. |
Resource borrowing and reclaiming work as follows:
-
A pod is scheduled if its requested resources, added to the node's current usage, stay within
max. -
If the node's usage exceeds
min, the excess is borrowed from idle capacity elsewhere in the tree. -
When another quota node needs its
minresources back, the scheduler selects pods from the borrowing node to evict based on factors such as job priority, availability, and creation time.
Features
-
Hierarchical quotas: Configure elastic quotas at multiple levels (for example, matching your organization structure). Each leaf node can map to multiple namespaces, but each namespace belongs to only one leaf node.
-
Resource borrowing and reclaiming: Idle
minresources can be borrowed by other quota nodes. Borrowed resources are reclaimed automatically when the original owner needs them.
-
Extended resource support: Beyond CPU and memory, capacity scheduling supports GPU (
nvidia.com/gpu) and other Kubernetes-supported resource types. -
Node affinity with ResourceFlavor: Attach a ResourceFlavor to a quota node to restrict pods in that quota to specific nodes. See Configure ResourceFlavor for node affinity.
Configure capacity scheduling
This example uses a cluster with a single ecs.sn2.13xlarge node (56 vCPUs and 224 GiB of memory).
Step 1: Create namespaces
kubectl create ns namespace1
kubectl create ns namespace2
kubectl create ns namespace3
kubectl create ns namespace4Step 2: Create an ElasticQuotaTree
Create the ElasticQuotaTree in the kube-system namespace. This example uses a two-level hierarchy with four leaf quota nodes.
ElasticQuotaTree only takes effect when created in the kube-system namespace.
apiVersion: scheduling.sigs.k8s.io/v1beta1
kind: ElasticQuotaTree
metadata:
name: elasticquotatree
namespace: kube-system
spec:
root:
name: root
max:
cpu: 40
memory: 40Gi
nvidia.com/gpu: 4
min:
cpu: 40
memory: 40Gi
nvidia.com/gpu: 4
children:
- name: root.a
max:
cpu: 40
memory: 40Gi
nvidia.com/gpu: 4
min:
cpu: 20
memory: 20Gi
nvidia.com/gpu: 2
children:
- name: root.a.1
namespaces:
- namespace1
max:
cpu: 20
memory: 20Gi
nvidia.com/gpu: 2
min:
cpu: 10
memory: 10Gi
nvidia.com/gpu: 1
- name: root.a.2
namespaces:
- namespace2
max:
cpu: 20
memory: 40Gi
nvidia.com/gpu: 2
min:
cpu: 10
memory: 10Gi
nvidia.com/gpu: 1
- name: root.b
max:
cpu: 40
memory: 40Gi
nvidia.com/gpu: 4
min:
cpu: 20
memory: 20Gi
nvidia.com/gpu: 2
children:
- name: root.b.1
namespaces:
- namespace3
max:
cpu: 20
memory: 20Gi
nvidia.com/gpu: 2
min:
cpu: 10
memory: 10Gi
nvidia.com/gpu: 1
- name: root.b.2
namespaces:
- namespace4
max:
cpu: 20
memory: 20Gi
nvidia.com/gpu: 2
min:
cpu: 10
memory: 10Gi
nvidia.com/gpu: 1The ElasticQuotaTree must satisfy these constraints:
-
Within each quota node:
min≤max -
For each parent node: sum of children's
minvalues ≤ parent'sminvalue -
For the root node:
min=max≤ total cluster resources -
Each namespace belongs to exactly one leaf node; a leaf node can contain multiple namespaces
Step 3: Verify the ElasticQuotaTree
kubectl get ElasticQuotaTree -n kube-systemExpected output:
NAME AGE
elasticquotatree 68sObserve resource borrowing and reclaiming
These scenarios show how borrowing and reclaiming work as workloads are deployed across the four namespaces.
Configure ResourceFlavor for node affinity
ResourceFlavor is a Kueue CRD that binds a quota node to specific nodes by matching node labels.
Prerequisites
Ensure that you have:
-
The ResourceFlavor CRD is applied (not installed by default)
-
kube-scheduler version higher than 6.9.0 (release notes, upgrade)
Only the nodeLabels field takes effect in ResourceFlavor.
Create a ResourceFlavor
This example creates a ResourceFlavor named spot that targets nodes labeled instance-type: spot.
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
name: "spot"
spec:
nodeLabels:
instance-type: spotAssociate a ResourceFlavor with an elastic quota
To bind a ResourceFlavor to a quota node, declare it in the ElasticQuotaTree using the attributes.resourceflavors field.
apiVersion: scheduling.sigs.k8s.io/v1beta1
kind: ElasticQuotaTree
metadata:
name: elasticquotatree
namespace: kube-system
spec:
root:
name: root
max:
cpu: 999900
memory: 400000Gi
nvidia.com/gpu: 100000
min:
cpu: 999900
memory: 400000Gi
nvidia.com/gpu: 100000
children:
- name: child
namespaces:
- default
attributes:
resourceflavors: spot
max:
cpu: 99
memory: 40Gi
nvidia.com/gpu: 10
min:
cpu: 99
memory: 40Gi
nvidia.com/gpu: 10With this configuration, pods in the child quota node (namespace default) are scheduled only to nodes with the instance-type: spot label.
Next steps
-
kube-scheduler also supports gang scheduling, which schedules all pods in a group together — if any pod cannot be scheduled, none are. Suited for big data workloads such as Spark and Hadoop. See Work with gang scheduling.