Alibaba Cloud Container Compute Service (ACS) provides the Gang Scheduling feature for jobs that require all-or-nothing scheduling.
Prerequisites
kube-scheduler installed at a version that meets the following requirements:
ACS cluster version Minimum scheduler version 1.31 v1.31.0-aliyun-1.2.0 1.30 v1.30.3-aliyun-1.1.1 1.28 v1.28.9-aliyun-1.1.0 -
Gang Scheduling supports only the GPU-HPN compute class. For more information, see Compute types.
-
The Enable custom labels and schedulers for GPU-HPN nodes setting must be disabled. For more information, see component configuration.
How it works
A job typically creates multiple pods that must start and run in a coordinated manner. This requires the scheduler to allocate resources to a group of pods as a single unit. This ensures that either all pods in the group acquire resources, or the entire group fails to schedule if even one pod cannot be placed. This all-or-nothing scheduling prevents resource deadlocks that occur when multiple jobs compete for resources.
ACS's built-in scheduler uses Gang Scheduling to implement these all-or-nothing semantics and ensure your jobs run smoothly.
All pods in a Gang Scheduling group must belong to the same compute-class.
Procedure
ACS's Gang Scheduling is compatible with the Kubernetes community PodGroup custom resource, version podgroups.scheduling.sigs.k8s.io/v1alpha1. Before you submit a job, you must create a PodGroup object in the job's namespace. In the PodGroup definition, you specify the minimum number of pods the job requires by setting the minMember field. Then, when you create the job's pods, you associate the pods with the PodGroup by adding the pod-group.scheduling.sigs.k8s.io label and setting its value to the PodGroup's name. The ACS scheduler then allocates resources to all pods with the same PodGroup label as a single unit.
-
Create the PodGroup custom resource.
apiVersion: scheduling.sigs.k8s.io/v1alpha1 kind: PodGroup metadata: name: demo-job-podgroup namespace: default spec: scheduleTimeoutSeconds: 10 minMember: 3 # Specify the minimum number of pods required to run. -
Create a job and associate it with the PodGroup.
apiVersion: batch/v1 kind: Job metadata: name: gang-job namespace: default spec: parallelism: 3 # The number of pods must be greater than or equal to minMember in the PodGroup. template: metadata: labels: alibabacloud.com/compute-class: "gpu-hpn" # Specify the compute-class as gpu-hpn. alibabacloud.com/gpu-model-series: "example-model" # Specify the GPU model for the GPU compute class. pod-group.scheduling.sigs.k8s.io: demo-job-podgroup # Associate with the demo-job-podgroup PodGroup. spec: containers: - name: demo-job image: registry.cn-hangzhou.aliyuncs.com/acs/stress:v1.0.4 args: - 'infinity' command: - sleep resources: requests: cpu: "1" memory: "1Gi" nvidia.com/gpu: "1" limits: cpu: "1" memory: "1Gi" nvidia.com/gpu: "1" restartPolicy: Never backoffLimit: 4
Ensure that the job's parallelism is greater than or equal to the minMember value in the PodGroup. Otherwise, the job will not be scheduled.
Example
This example demonstrates both a failed and a successful scheduling scenario when using Gang Scheduling.
-
Run the following command to create the
test-gangnamespace.kubectl create ns test-gang -
Run the following command to create a ResourceQuota in the
test-gangnamespace. This simulates a scenario with insufficient resources to demonstrate how Gang Scheduling works.cat << EOF | kubectl apply -f - apiVersion: v1 kind: ResourceQuota metadata: name: object-counts namespace: test-gang spec: hard: pods: "2" EOF -
Run the following command to create a PodGroup object. The
minMemberfield is set to 3, which means at least three pods from the group must be scheduled simultaneously. If this condition is not met, the other pods will remain in the Pending state.cat << EOF | kubectl apply -f - apiVersion: scheduling.sigs.k8s.io/v1alpha1 kind: PodGroup metadata: name: demo-job-podgroup namespace: test-gang spec: minMember: 3 # Specify the minimum number of pods required to run. EOF -
Create a file named gang-job.yaml with the following content. This file defines a job object with four pod replicas and associates it with the PodGroup object.
apiVersion: batch/v1 kind: Job metadata: name: gang-job namespace: test-gang spec: parallelism: 4 # The number of pods must be greater than or equal to minMember in the PodGroup. template: metadata: labels: alibabacloud.com/compute-class: "gpu-hpn" # Specify the compute-class as gpu-hpn. alibabacloud.com/gpu-model-series: "example-model" # Specify the GPU model for the GPU compute class. pod-group.scheduling.sigs.k8s.io: demo-job-podgroup # Associate with the demo-job-podgroup PodGroup. spec: containers: - name: demo-job image: registry.cn-hangzhou.aliyuncs.com/acs/stress:v1.0.4 args: - 'infinity' command: - sleep resources: requests: cpu: "1" memory: "1Gi" nvidia.com/gpu: "1" limits: cpu: "1" memory: "1Gi" nvidia.com/gpu: "1" restartPolicy: Never backoffLimit: 4 -
Run the following command to deploy the gang-job to the cluster.
kubectl apply -f gang-job.yaml -
Run the following command to check the pod status.
kubectl get pod -n test-gangExpected output:
NAME READY STATUS RESTARTS AGE gang-job-hrnc6 0/1 Pending 0 23s gang-job-wthnq 0/1 Pending 0 23sThe ResourceQuota limits the namespace to a maximum of two pods, which is less than the
minMembervalue specified in the PodGroup. As a result, these two pods remain in the Pending state and are not scheduled. -
Run the following command to delete the ResourceQuota and remove the pod limit.
kubectl delete resourcequota -n test-gang object-counts -
Run the following command to check the pod status again.
kubectl get pod -n test-gangExpected output:
NAME READY STATUS RESTARTS AGE gang-job-24cz9 1/1 Running 0 96s gang-job-mmkxl 1/1 Running 0 96s gang-job-msr8v 1/1 Running 0 96s gang-job-qnclz 1/1 Running 0 96sThe output shows that the pods are scheduled successfully.