Configure concurrency limits for a pipeline to control how many pipeline instances and task instances can run simultaneously.
Example
concurrencyConfig:
maxRunningPipelineInstances: 1
pipelineMaxRunningBehavior: preemptOldest
maxRunningJobInstances: 100Details
concurrencyConfig
Controls the concurrent execution behavior of the pipeline and its tasks.
concurrencyConfig.maxRunningPipelineInstances
Optional. The maximum number of pipeline instances that can run concurrently. Default value: 100.
When the maximum number of running instances is reached, pipelineMaxRunningBehavior determines how new execution requests are handled.
concurrencyConfig.pipelineMaxRunningBehavior
Optional. Default value: rejectNew.
Specifies the behavior when the maximum number of running instances is reached. Valid values:
-
preemptOldest: Cancels the oldest running instance to make room for the new one. -
rejectNew: Denies new execution requests.
concurrencyConfig.maxRunningJobInstances
Optional. The maximum number of instances of the same job that can run concurrently. Default value: 1.
When multiple pipeline instances run concurrently, this setting limits the number of parallel executions for the same job. If the limit is exceeded, new task instances wait for existing ones to complete.
Scenarios
-
Limit pipeline concurrency: To limit the number of pipeline instances that can run concurrently, set
maxRunningPipelineInstances. -
Control job concurrency: To allow the same job to run concurrently across different pipeline instances, set
maxRunningJobInstances. -
Preemption policy: To prioritize newly triggered pipelines, set
pipelineMaxRunningBehaviortopreemptOldest. -
Rejection policy: To strictly limit concurrency and prevent new instances from preempting existing ones, set
pipelineMaxRunningBehaviortorejectNew.