DataWorks Data Integration lets you migrate all or selected tables from a source database to a destination store through full or incremental synchronization, on a one-time or recurring schedule. It automatically creates destination table schemas, eliminating the need to configure individual tasks per table.
Use cases
Data migration and cloud adoption
Migrate databases, such as MySQL and Oracle, from an on-premises data center to a cloud data warehouse or data lake.
Migrate data between different cloud platforms or database systems.
Data warehouse or data lake construction
Periodically synchronize full or incremental data from OLTP databases to the ODS layer of a data warehouse or data lake for downstream analytics.
Data backup and disaster recovery
Regularly back up full datasets from production databases to cost-effective storage media, such as HDFS or OSS.
Implement cross-region or cross-availability-zone disaster recovery solutions.
Core capabilities
The following table describes the core capabilities.
Core capability | Feature | Description |
Synchronization across heterogeneous data sources | - | Batch database synchronization supports migrating data from on-premises data centers or other cloud platforms to data warehouses or data lakes such as MaxCompute, Hologres, and OSS. For more information, see Supported data sources and destinations. |
Data synchronization in complex network environments | - | Offline synchronization supports data synchronization across Alibaba Cloud databases, on-premises IDC databases, self-managed databases on ECS instances, and non-Alibaba Cloud databases. Before configuration, make sure that the resource group has network connectivity with the source and destination. For more information, see Configure network connectivity. |
Synchronization scenarios | Full synchronization | Supports one-time or recurring full data synchronization to destination tables or specified partitions. |
Incremental synchronization | Supports one-time or recurring incremental data synchronization based on time, partitions, or primary keys. | |
Combined full and incremental synchronization | First run: Automatically performs a full data synchronization. Subsequent runs: Automatically switches to recurring incremental data synchronization to specified partitions. | |
Database and table mapping | Batch table synchronization | You can synchronize all tables in a database, or select specific tables by manually choosing them or configuring filter rules. |
Automatic table creation | With a single configuration, you can process hundreds of tables from the source database. The system automatically creates table schemas at the destination without manual intervention. | |
Flexible mapping | Supports custom naming rules for destination databases and tables, and custom field type mappings between the source and destination to flexibly adapt to the data structure model of the destination. | |
Scheduling and dependency management | Schedule | Supports multiple scheduling intervals, such as minute, hour, day, week, month, and year. If you need to synchronize a large number of tables at a time, we recommend that you configure schedules to run tasks in batches to prevent task accumulation and resource contention. |
Task dependencies | Both the batch database task and each table-level subtask can serve as upstream dependencies in DataWorks, allowing other development tasks to depend on them. When the synchronization task for a specific table is completed, its downstream development tasks are automatically triggered. | |
Parameter support | Supports scheduling parameters for incremental synchronization. For example, you can use ${bizdate} to represent the business date. | |
Advanced parameters | Dirty data settings | Dirty data refers to data records that fail to be written to the destination due to exceptions such as type conflicts or constraint violations. The default value is false, which means dirty data is not allowed. If dirty data occurs, the task fails. If you set this parameter to true, all dirty data is ignored. |
Reader and writer settings | Supports configuring the maximum number of connections for the reader and writer data sources separately, and defining cleanup policies for the destination before data is written. | |
Concurrency and throttling |
| |
OM | Online intervention | Supports online intervention operations such as rerun, backfill, mark as successful, and freeze/unfreeze. |
Monitoring and alerting | Supports configuring monitoring rules based on baselines, task statuses, and running durations, and supports alerting for triggered rules. | |
Data quality | After a task is committed and deployed, you can configure data quality monitoring rules for destination tables in Operation Center. AI-based automatic generation and manual configuration are both supported. Currently, only certain database types support quality rule monitoring. For more information, see Data quality. | |
DI Agent | AI Native | DI Agent is a vertical capability set of Data Agent for data integration scenarios. It covers six capabilities: conversational task configuration, multimodal data processing, ChatDB data Q&A, intelligent diagnostics, periodic inspection, and IM channel integration. |
Get started
To create a batch database synchronization task, see Create a batch database synchronization task.
To create and manage batch database synchronization tasks through natural language conversations, you can use the Data Integration AI Native capability (DI Agent). For more information, see Use DI Agent.
Supported data sources
DataWorks supports batch migration of data from various data sources to destinations such as MaxCompute, OSS, and Elasticsearch. The following data source types are supported.
Source data source | Destination data source |
MaxCompute | |
Data Lake Formation | |
Hive | |
Hologres | |
OSS | |
OSS-HDFS | |
Elasticsearch | |
StarRocks | |
MySQL |