Task orchestration overview
Task orchestration in Data Management (DMS) lets you build task flows from one or more task nodes and run them on timed or event-based schedules to automate complex data workflows.
Background information
As data volumes grow and sources diversify, manual data processing no longer scales. Automated workflows are essential to ensure correct, repeatable data processing, analysis, and backup.
Task orchestration in DMS automates these processes, reduces errors, improves data development efficiency, and enhances data value and reliability.
Supported database types
-
Relational databases
-
MySQL: RDS MySQL, PolarDB for MySQL, MyBase MySQL, PolarDB Distributed Edition, and other MySQL sources
-
SQL Server: RDS SQL Server, MyBase SQL Server, and SQL Server from other sources
-
PostgreSQL: RDS PostgreSQL, PolarDB for PostgreSQL, MyBase PostgreSQL, and PostgreSQL from other sources
-
OceanBase: OceanBase MySQL mode, OceanBase Oracle mode, or self-managed OceanBase
-
PolarDB for PostgreSQL (Compatible with Oracle)
-
Oracle
-
Dameng Database
-
DB2
-
-
NoSQL: Lindorm
-
Data warehouses
-
AnalyticDB for MySQL
-
AnalyticDB for PostgreSQL
-
DLA
-
MaxCompute
-
Hologres
-
-
Object Storage Service: OSS
Job types
|
Category |
Task Type |
Description |
References |
|
Data integration |
DTS data migration |
Migrates data and schemas across databases at the table or database level. |
|
|
Offline data integration |
Synchronizes data between sources for migration and transmission scenarios. |
||
|
Data transformation |
Single Instance SQL |
Runs SQL statements on a specified relational database. Note
If lock-free schema change is enabled for the target instance, DMS preferentially applies this feature when you run Single Instance SQL tasks to prevent table locks. Enable the lock-free schema change feature. |
N/A |
|
Loop SQL |
Generates SQL statements in a loop and submits them in batches to any DMS-managed relational database. Variables control loop count to simplify logic and reduce database load. |
||
|
Cross-database Spark SQL |
Uses the Spark engine for large-scale, cross-database data synchronization and transformation. |
||
|
Lindorm Spark |
Uses the Lindorm compute engine for distributed computing tasks such as data production, interactive analysis, machine learning, and graph computing. |
||
|
ADB Spark |
Runs Spark jobs on periodic, sequential schedules. |
||
|
DLA Serverless Spark |
Configures Serverless Spark jobs. |
||
|
Lock-free data change |
Performs UPDATE and DELETE operations using DMS lock-free technology. Note
Lock-free schema change must be enabled for the target instance. Enable the lock-free schema change feature. |
||
|
DLA Spark SQL |
Submits jobs to a DLA Spark cluster using SQL. |
None |
|
|
General |
Single-instance SQL assignment |
Assigns data read by a SELECT statement to an output variable for use in the next node. |
|
|
Conditional branch |
Evaluates a conditional expression in a task flow. If the expression is true, subsequent tasks run; otherwise, they are skipped. |
||
|
DLA one-click data warehouse |
Uploads data from a database to OSS and builds a data warehouse in one step. |
||
|
DBS backup |
Backs up database data to the OSS provided by DBS. |
||
|
EC remote command |
Runs Shell, PowerShell, or Bat commands or scripts on a remote ECS instance through Cloud Assistant. |
||
|
Script code |
Runs script jobs in Database Gateway on periodic or scheduled intervals. |
||
|
Dashboard push |
Pushes DataService Studio visualization charts to DingTalk, DingTalk bots, or email using filter interactions. |
||
|
Status check |
Table time column status check |
Checks whether new data exists in the target table after a specified time. |
None |
|
Lindorm file check |
Checks whether a file exists in Lindorm HDFS. |
None |
|
|
SQL status check |
Validates data status using an SQL statement, such as checking whether a table meets a row-count threshold. |
None |
|
|
Audit task |
Checks Data Quality. After you set quality rules and a scheduling cycle, the task automatically checks Data Quality and generates a report. |
None |
|
|
Task flow dependency check |
Supports self-dependencies within a task stream and dependencies between task streams. The dependency target can be a task stream or a task node. |