Data Management (DMS) enforces the following limits on task orchestration and data warehouse development features.
Quantity limits
| Resource | Limit |
|---|---|
| Task flows per DMS tenant (task orchestration + data warehouse development combined) | 5,000 |
| Task nodes per task flow | 100 |
| Business scenarios (task orchestration) | 200 |
Duration limits
Task flow
A task flow cannot run longer than 18 hours.
If Enable re-run of failed task is turned on in the Advanced Settings tab, a failed task node is rerun after a delay. The total duration includes time spent before the rerun, so the task flow may exceed 18 hours in total.
Task node
Each task node must complete within the duration limit for its type. If a task node exceeds its limit, it is forcibly stopped and fails.
Task node type | Task node name | Execution duration limit |
Data integration | DTS data migration | About 14 hours. |
Offline data integration | About 8 hours. | |
Data processing | Single-instance SQL | About 14 hours. |
Loop SQL | About 14 hours for a single batch of SQL statements. The total duration is limited by the 18-hour execution duration limit of the task flow. | |
Cross-database Spark SQL | About 8 hours. | |
Lindorm Spark | About 11 hours. | |
ADB Spark | About 14 hours. | |
DLA Serverless Spark | About 5 hours and 30 minutes. | |
Lock-free schema change | About 14 hours. | |
DLA Spark SQL | About 5 hours and 30 minutes. | |
General | Single-instance SQL Assignment | The execution duration depends on the query timeout period configured for the instance. |
Conditional Branch | The task node execution is extremely short. | |
DLA one-click data warehousing | About 5 hours and 30 minutes. | |
DBS Backup | No execution duration limit. | |
ECS Remote Command | About 11 hours. | |
Script Command | About 18 hours. | |
Status Check | Table Time Column Status Check | The task node execution is extremely short. |
Lindorm File Check | The task node execution is extremely short. | |
SQL Status Check | The execution duration depends on the query timeout period configured for the instance. | |
Audit Task | The task node execution is extremely short. | |
Task Flow Dependency Check | The task node execution is extremely short. |
Usage notes
Scheduling latency
Task scheduling latency may occur during peak hours, such as early morning and the start of each hour. Account for scheduling latency when writing SQL statements to make sure output data remains correct if a delay occurs.
Idempotent SQL statements
Idempotence means that running an SQL statement or script multiple times produces the same result as running it once.
Write idempotent SQL statements and scripts — for example, using insert overwrite or replace into. This ensures output data is correct even if a task node is retried. Run your statements multiple times before deploying to verify they behave as expected.