Set Slurm queue priorities-Container Service for Kubernetes(ACK)-阿里云帮助中心

Choose a preemption strategy

Use this table to select the right strategy for your use case:

Strategy	Key parameter	Best for
Partition priority	`PreemptType=preempt/partition_prio`	Separating urgent from non-urgent workloads into distinct partitions
QoS preemption	`PreemptType=preempt/qos`	Fine-grained priority control across users or job classes
Job size priority	`PriorityFavorSmall=YES`	Mixed small/large job clusters where runtimes are unpredictable

All three strategies require this base configuration in slurm.conf:

Parameter	Recommended value	Description
`SelectType`	`select/cons_tres`	Resource allocation strategy. Dynamic node workers require `select/cons_tres`.
`SelectTypeParameters`	`CR_Core`	Resource allocation details for the `SelectType` plugin.
`SchedulerType`	`sched/backfill`	Scheduling algorithm. Backfill scheduling is enabled by default.
`PriorityType`	`priority/multifactor`	Priority calculation method.
`PreemptMode`	`SUSPEND,GANG`	Preemption behavior. With `select/cons_tres`, only `SUSPEND` and `GANG` are supported.

How Slurm evaluates and schedules jobs

Understanding how the scheduler selects jobs helps you predict the effect of your configuration.

Scheduling behavior is controlled by SchedulerType (default: sched/backfill) and fine-tuned through SchedulerParameters in slurm.conf. See the Slurm scheduling configuration guide.

Queue types

Slurm supports two priority mechanisms:

Queue type	How it works	When to use
First In, First Out (FIFO)	Jobs run in submission order (`priority/basic`)	Simple clusters where submission order is sufficient
Multifactor	Job priority is a weighted sum of multiple factors	Clusters with diverse workloads or fairness requirements

Multifactor scheduling is enabled by default. If your cluster runs uniform workloads with no preemption requirements, FIFO may be sufficient.

FIFO queue

Set PriorityType=priority/basic in slurm.conf to enable FIFO scheduling:

# Back up slurm.conf before editing
sudo cp /etc/slurm-llnl/slurm.conf /etc/slurm-llnl/slurm.conf.bak

sudo nano /etc/slurm-llnl/slurm.conf
# Add or update:
PriorityType=priority/basic

Important

Back up slurm.conf before making changes. Test in a non-production environment first.

Multifactor queue

Multifactor scheduling computes each job's priority as a weighted sum:

Job_priority =
  site_factor +
  (PriorityWeightAge)       * (age_factor) +
  (PriorityWeightAssoc)     * (assoc_factor) +
  (PriorityWeightFairshare) * (fair-share_factor) +
  (PriorityWeightJobSize)   * (job_size_factor) +
  (PriorityWeightPartition) * (priority_job_factor) +
  (PriorityWeightQOS)       * (QOS_factor) +
  SUM(TRES_weight_cpu * TRES_factor_cpu,
      TRES_weight_<type> * TRES_factor_<type>,
      ...)
  - nice_factor

Each weight controls a different scheduling behavior:

Factor	Weight parameter	Effect
Waiting time	`PriorityWeightAge`	Longer-waiting jobs score higher
Association	`PriorityWeightAssoc`	Adjusts score by user group or account
Fair-share	`PriorityWeightFairshare`	Boosts jobs from users who have consumed fewer resources
Job size	`PriorityWeightJobSize`	Favors small or large jobs depending on configuration
Partition	`PriorityWeightPartition`	Raises score for jobs in designated partitions
QoS	`PriorityWeightQOS`	Applies Quality of Service level scoring
Trackable RESources (TRES)	`TRES_weight_<type>`	Weights specific resource types (CPU, GPU, and others)
Nice	—	Subtracts from priority; a higher `nice` value lowers priority

See the multifactor priority plugin documentation.

Common configurations:

Prioritize small jobs: Set PriorityWeightJobSize=-1 to lower large job priority and reduce head-of-line blocking.
Protect critical teams: Tune PriorityWeightAssoc and fair-share_factor to ensure jobs from important accounts run first.
Prevent starvation: Set PriorityWeightFairshare=2000 to boost jobs from users with low historical usage.

Configure partition priority preemption

Create two partitions over the same node pool—one for urgent jobs, one for non-urgent. Submit or move a job to the high-priority partition to preempt lower-priority work. Preempted jobs are suspended or canceled based on your PreemptMode setting.

Prerequisites

Before you begin, ensure that you have:

A running Slurm cluster on ACK with a worker node pool
Root or sudo access to edit slurm.conf
A non-production environment for testing changes

Set up partition-based preemption

Edit slurm.conf to enable preemption:

Important

Back up slurm.conf before editing. Test changes in a non-production environment first.

sudo nano /etc/slurm-llnl/slurm.conf

Add or update these parameters:

# Preemption based on partition priority tier
PreemptType=preempt/partition_prio

# Behavior when a job is preempted:
# suspend — pauses the job until resources are available again
# cancel  — terminates the job immediately
PreemptMode=SUSPEND,GANG

Create a high-priority partition:
```
scontrol create partition=hipri PriorityTier=2 nodes=ALL
```
Alternatively, add the partition directly to slurm.conf.

Verify both partitions are visible:

scontrol show partition

Expected output (key fields shown):

PartitionName=debug
   PriorityTier=1 OverSubscribe=FORCE:1
   PreemptMode=GANG,SUSPEND
   State=UP TotalCPUs=4 TotalNodes=1

PartitionName=hipri
   PriorityTier=2 OverSubscribe=NO
   PreemptMode=GANG,SUSPEND
   State=UP TotalCPUs=0 TotalNodes=0

Trigger preemption

Submit a job to the high-priority partition to preempt a running low-priority job:

# Submit four jobs to the default partition
srun sleep 1d &
srun sleep 1d &
srun sleep 1d &
srun sleep 1d &

# Confirm all four are running
squeue

JOBID  PARTITION   NAME     USER   ST     TIME  NODES NODELIST(REASON)
    4     debug    sleep     root  R       0:03  1    slurm-test-worker-cpu-0
    3     debug    sleep     root  R       0:04  1    slurm-test-worker-cpu-0
    2     debug    sleep     root  R       0:04  1    slurm-test-worker-cpu-0
    1     debug    sleep     root  R       0:05  1    slurm-test-worker-cpu-0

# Submit a job to the high-priority partition
srun --partition=hipri sleep 1d &
squeue

Job 4 is suspended (ST changes from R to S), and job 5 is running:

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
    5     hipri    sleep     root  R       0:06      1 slurm-test-worker-cpu-0
    4     debug    sleep     root  S       0:59      1 slurm-test-worker-cpu-0
    3     debug    sleep     root  R       1:06      1 slurm-test-worker-cpu-0
    2     debug    sleep     root  R       1:06      1 slurm-test-worker-cpu-0
    1     debug    sleep     root  R       1:07      1 slurm-test-worker-cpu-0

To escalate a pending job without resubmitting it, update its partition:

scontrol update jobid=6 partition=hipri
squeue

Jobs 1 and 2 are suspended—jobs in hipri share execution time via gang scheduling:

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
    6     hipri    sleep     root  R       0:03      1 slurm-test-worker-cpu-0
    5     hipri    sleep     root  R       3:33      1 slurm-test-worker-cpu-0
    4     debug    sleep     root  R       3:21      1 slurm-test-worker-cpu-0
    3     debug    sleep     root  R       3:33      1 slurm-test-worker-cpu-0
    2     debug    sleep     root  S       3:41      1 slurm-test-worker-cpu-0
    1     debug    sleep     root  S       4:01      1 slurm-test-worker-cpu-0

Configure QoS-based preemption

QoS preemption assigns priority levels to jobs regardless of partition. A high-priority QoS job preempts lower-priority ones. With PreemptMode=SUSPEND,GANG, preempted jobs are suspended and time-share resources rather than being terminated.

Slurm includes a normal QoS (priority 0) by default.

Prerequisites

Before you begin, ensure that you have:

A running Slurm cluster on ACK
Root or sudo access to edit slurm.conf and run sacctmgr
Base preemption parameters set in slurm.conf (see Choose a preemption strategy)

Set up QoS-based preemption

Enable QoS preemption in slurm.conf:
```
sudo nano /etc/slurm-llnl/slurm.conf
```
Add or update:
```
PreemptType=preempt/qos
PreemptMode=SUSPEND,GANG
```
Also add OverSubscribe=FORCE:1 to each affected partition definition in slurm.conf.

Create a high-priority QoS:

sacctmgr add qos high preempt=normal preemptmode=gang,suspend priority=10

Enter y when prompted by sacctmgr. Parameter meanings:

Parameter	Value	Effect
`preempt`	`normal`	Allows `high` QoS jobs to preempt `normal` QoS jobs
`preemptmode`	`gang,suspend`	Gang: preempting job acquires all resources first. Suspend: preempted jobs pause and resume afterward.
`priority`	`10`	Base priority score for `high` QoS jobs (higher = higher priority)

Verify the QoS configuration:

sacctmgr show qos format=name,priority,preempt

Expected output:

      Name   Priority    Preempt
---------- ---------- ----------
    normal          0
      high         10     normal

Trigger QoS preemption

# Submit five jobs with normal QoS
sbatch test.sh   # job 4
sbatch test.sh   # job 5
sbatch test.sh   # job 6
sbatch test.sh   # job 7
sbatch test.sh   # job 8
squeue

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
    8     debug  test.sh     root PD       0:00      1 (Resources)
    7     debug  test.sh     root  R       0:03      1 slurm-test-worker-cpu-0
    6     debug  test.sh     root  R       0:15      1 slurm-test-worker-cpu-0
    5     debug  test.sh     root  R       0:15      1 slurm-test-worker-cpu-0
    4     debug  test.sh     root  R       0:18      1 slurm-test-worker-cpu-0

# Submit a high-priority QoS job
sbatch --qos=high test.sh   # job 9
squeue

Job 9 runs immediately, time-sharing resources with other jobs:

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
    9     debug  test.sh     root  S       0:00      1 slurm-test-worker-cpu-0
    8     debug  test.sh     root PD       0:00      1 (Resources)
    7     debug  test.sh     root  R       0:26      1 slurm-test-worker-cpu-0
    6     debug  test.sh     root  R       0:38      1 slurm-test-worker-cpu-0
    5     debug  test.sh     root  R       0:38      1 slurm-test-worker-cpu-0
    4     debug  test.sh     root  R       0:41      1 slurm-test-worker-cpu-0

Configure job size priority

When job runtimes are unpredictable, job size priority reduces head-of-line blocking by favoring small jobs. Waiting time accumulates to prevent large jobs from starving.

How scoring works

When PriorityFavorSmall=YES, the job size score is:

size_score = (1 - requested_resources / total_cluster_resources) x PriorityWeightJobSize

Example with a 4-CPU cluster and PriorityWeightJobSize=1000:

Job size	Score calculation	Score
1 CPU	(1 - 1/4) x 1000	750
4 CPUs	(1 - 4/4) x 1000	0

The age factor accumulates. With PriorityWeightAge=1000 and PriorityMaxAge=1-0 (1 day):

Each minute of waiting adds approximately 0.69 points.
After 24 hours, the job receives the full 1,000 age points.
Jobs exceeding PriorityMaxAge receive the full PriorityWeightAge score.

This prevents large jobs from starving: the age factor eventually outweighs the size penalty.

Set up job size priority

Add the following to slurm.conf:

PriorityFavorSmall=YES
PriorityWeightAge=1000
PriorityWeightJobSize=1000
PriorityMaxAge=1-0

If job runtimes are estimable, enable backfill scheduling (on by default):

SchedulerType=sched/backfill

Backfill fills idle slots with small jobs that finish without delaying larger ones.

Next steps

Slurm multifactor priority plugin — full weight configuration reference
Slurm scheduling configuration — backfill and scheduler parameters