Use Gang scheduling to solve All-or-Nothing job scheduling issues-Container Service for Kubernetes(ACK)-阿里云帮助中心

How it works

ACK implements gang scheduling through a PodGroup resource. Each pod is assigned to a PodGroup with a minimum pod count (min-available). The scheduler holds the group in Pending until the minimum is met, then dispatches all pods together.

ACK supports three methods to define a PodGroup:

Method	How pods are grouped	Best for
Labels	kube-scheduler auto-creates a PodGroup	Simple jobs, no separate PodGroup object needed
PodGroup CRD	Explicit PodGroup resource with timeout control	Jobs that need `scheduleTimeoutSeconds`
Koordinator annotations	Annotation-based grouping	Clusters using the Koordinator scheduling stack

All methods require pods and their PodGroup in the same namespace. All pods in a PodGroup must share the same priority.

Prerequisites

Before you begin, ensure that you have:

An ACK managed Pro cluster running Kubernetes 1.16 or later. Upgrade the cluster if needed.
For advanced configurations (GangGroup and match policy): a cluster running Kubernetes 1.22 or later with kube-scheduler version later than 1.xx.xx-aliyun-4.0.

Ensure that the elastic node pool has sufficient maximum resource capacity and matching node labels for pod scheduling. Otherwise, pods may fail to use nodes scaled out by the elastic node pool.

Enable gang scheduling

Method 1: Labels (recommended)

Add two labels to each pod. kube-scheduler automatically creates a PodGroup named after the pod-group.scheduling.sigs.k8s.io/name value.

labels:
  pod-group.scheduling.sigs.k8s.io/name: tf-smoke-gpu       # PodGroup name (must be a valid DNS subdomain)
  pod-group.scheduling.sigs.k8s.io/min-available: "3"       # Minimum pods required to start the job

The pod-group.scheduling.sigs.k8s.io/name value must be a valid DNS subdomain name.

Method 2: PodGroup CRD

Create a PodGroup resource, then reference it from each pod with a label.

Important

Since ACK 1.31, only scheduling.x-k8s.io/v1alpha1 is supported. scheduling.sigs.k8s.io/v1alpha1 is no longer supported.

# PodGroup resource
apiVersion: scheduling.x-k8s.io/v1alpha1
kind: PodGroup
metadata:
  name: nginx
spec:
  scheduleTimeoutSeconds: 10   # Seconds to wait before rejecting the group if min-available is not met
  minMember: 3                 # Minimum number of pods required to start the job
---
# Pod label — must match the PodGroup name and namespace
labels:
  pod-group.scheduling.sigs.k8s.io/name: nginx

Method 3: Koordinator annotations

Add annotations to each pod. This method does not support the total-number or mode parameters from the Koordinator API.

annotations:
  gang.scheduling.koordinator.sh/name: "gang-example"
  gang.scheduling.koordinator.sh/min-available: "2"

Advanced configurations

Group multiple gangs (GangGroup)

Some jobs have roles with different min-available requirements — for example, PyTorch jobs with a parameter server and workers. A single PodGroup cannot express per-role minimums, and separate PodGroups cannot coordinate across roles.

GangGroup links multiple PodGroups. The job starts only when every gang satisfies its own min-available. Add the following label to each pod or PodGroup (use the annotation key for the Koordinator method):

Method	Resource	Key
Labels	Pod	`pod-group.scheduling.sigs.k8s.io/groups`
PodGroup CRD	PodGroup	`pod-group.scheduling.sigs.k8s.io/groups`
Koordinator annotations	Pod	`gang.scheduling.koordinator.sh/groups`

Example value (JSON array of <namespace>/<gang-name> entries):

pod-group.scheduling.sigs.k8s.io/groups: "[\"default/gang-example1\", \"default/gang-example2\"]"

GangGroup requires Kubernetes 1.22 or later with kube-scheduler version later than 1.xx.xx-aliyun-4.0.

Configure match policy

By default, a PodGroup counts only pods that completed resource preallocation (only-waiting). Use match-policy to include pods in other states toward the minimum — for example, when pods from a prior cycle are still running.

Add the label to each pod (labels method) or to the PodGroup resource (PodGroup CRD method). The Koordinator annotations method only supports once-satisfied.

Match policy	Pods counted toward min-available	When to use
`only-waiting`	Pods that completed resource preallocation	Strictest. Use for stateless jobs with no carry-over from prior cycles.
`waiting-and-running`	Pods in Running state + pods that completed preallocation	Use when running pods from a prior cycle should count toward the minimum.
`waiting-running-succeed`	Pods in Succeeded state + Running + completed preallocation	Use for jobs that tolerate partial restarts — already-succeeded pods still count.
`once-satisfied`	Pods that completed resource preallocation; PodGroup becomes invalid once satisfied	Use for one-shot jobs. Once the gang is dispatched, the PodGroup is invalidated.

Labels-based example:

pod-group.scheduling.sigs.k8s.io/match-policy: "waiting-and-running"

PodGroup CRD example (add to the PodGroup, not the pod):

pod-group.scheduling.sigs.k8s.io/match-policy: "waiting-and-running"

Match policy configuration requires Kubernetes 1.22 or later with kube-scheduler version later than 1.xx.xx-aliyun-4.0.

Example: distributed TensorFlow job

This example compares a distributed TensorFlow job with and without gang scheduling. The cluster has 4 GPUs. The job has 1 PS pod and 4 worker pods, each requiring 2 GPUs, with min-available set to 5.

Step 1: Install Arena and prepare the cluster for TensorFlow jobs.

Arena is a Kubeflow subproject for managing machine learning job lifecycles — including environment setup, data preparation, model development, training, and prediction — through a CLI or SDK.

Step 2: Submit the TensorFlow job. Both the PS and worker templates include gang scheduling labels with min-available: "5".

apiVersion: "kubeflow.org/v1"
kind: "TFJob"
metadata:
  name: "tf-smoke-gpu"
spec:
  tfReplicaSpecs:
    PS:
      replicas: 1
      template:
        metadata:
          creationTimestamp: null
          labels:
            pod-group.scheduling.sigs.k8s.io/name: tf-smoke-gpu
            pod-group.scheduling.sigs.k8s.io/min-available: "5"
        spec:
          containers:
          - args:
            - python
            - tf_cnn_benchmarks.py
            - --batch_size=32
            - --model=resnet50
            - --variable_update=parameter_server
            - --flush_stdout=true
            - --num_gpus=1
            - --local_parameter_device=cpu
            - --device=cpu
            - --data_format=NHWC
            image: registry.cn-hangzhou.aliyuncs.com/kubeflow-images-public/tf-benchmarks-cpu:v20171202-bdab599-dirty-284af3
            name: tensorflow
            ports:
            - containerPort: 2222
              name: tfjob-port
            resources:
              limits:
                cpu: '1'
            workingDir: /opt/tf-benchmarks/scripts/tf_cnn_benchmarks
          restartPolicy: OnFailure
    Worker:
      replicas: 4
      template:
        metadata:
          creationTimestamp: null
          labels:
            pod-group.scheduling.sigs.k8s.io/name: tf-smoke-gpu
            pod-group.scheduling.sigs.k8s.io/min-available: "5"
        spec:
          containers:
          - args:
            - python
            - tf_cnn_benchmarks.py
            - --batch_size=32
            - --model=resnet50
            - --variable_update=parameter_server
            - --flush_stdout=true
            - --num_gpus=1
            - --local_parameter_device=cpu
            - --device=gpu
            - --data_format=NHWC
            image: registry.cn-hangzhou.aliyuncs.com/kubeflow-images-public/tf-benchmarks-gpu:v20171202-bdab599-dirty-284af3
            name: tensorflow
            ports:
            - containerPort: 2222
              name: tfjob-port
            resources:
              limits:
                nvidia.com/gpu: 2
            workingDir: /opt/tf-benchmarks/scripts/tf_cnn_benchmarks
          restartPolicy: OnFailure

Without gang scheduling enabled:

Check pod status:

kubectl get pods

With only 4 GPUs, 2 workers start Running and claim all GPUs. The remaining 2 stay Pending, blocking the job:

NAME                    READY   STATUS    RESTARTS   AGE
tf-smoke-gpu-ps-0       1/1     Running   0          6m43s
tf-smoke-gpu-worker-0   1/1     Running   0          6m43s
tf-smoke-gpu-worker-1   1/1     Running   0          6m43s
tf-smoke-gpu-worker-2   0/1     Pending   0          6m43s
tf-smoke-gpu-worker-3   0/1     Pending   0          6m43s

Check a running worker's logs:

kubectl logs -f tf-smoke-gpu-worker-0

Workers are stalled waiting for the Pending pods — GPUs held, no training runs:

INFO|2020-05-19T07:02:18|/opt/launcher.py|27| 2020-05-19 07:02:18.199696: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:worker/replica:0/task:3
INFO|2020-05-19T07:02:28|/opt/launcher.py|27| 2020-05-19 07:02:28.199798: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:worker/replica:0/task:2

With gang scheduling enabled:

All 5 pods remain Pending until the cluster has enough resources to satisfy min-available: 5:

NAME                    READY   STATUS    RESTARTS   AGE
tf-smoke-gpu-ps-0       0/1     Pending   0          43s
tf-smoke-gpu-worker-0   0/1     Pending   0          43s
tf-smoke-gpu-worker-1   0/1     Pending   0          43s
tf-smoke-gpu-worker-2   0/1     Pending   0          43s
tf-smoke-gpu-worker-3   0/1     Pending   0          43s

After 4 GPUs are added to the cluster, the scheduler dispatches all 5 pods simultaneously:

kubectl get pods

Expected output:

NAME                    READY   STATUS    RESTARTS   AGE
tf-smoke-gpu-ps-0       1/1     Running   0          3m16s
tf-smoke-gpu-worker-0   1/1     Running   0          3m16s
tf-smoke-gpu-worker-1   1/1     Running   0          3m16s
tf-smoke-gpu-worker-2   1/1     Running   0          3m16s
tf-smoke-gpu-worker-3   1/1     Running   0          3m16s

Check the worker log to confirm training has started:

kubectl logs -f tf-smoke-gpu-worker-0