DataWorks Data Integration provides a powerful real-time database synchronization solution that replicates all or selected tables from a source database to a destination data store with low latency in an automated, integrated full-and-incremental manner. Built on a real-time computing engine, this feature automatically performs the initial full data load and then seamlessly switches to continuous capture of incremental change data (CDC). It serves as a one-stop solution for use cases such as real-time cloud migration and real-time data warehouse ODS layer construction.
Use cases
Build a real-time ODS layer for your data warehouse
Synchronize data from operational databases such as MySQL and Oracle to a real-time data warehouse like Hologres or StarRocks, providing fresh data for dashboards and ad hoc queries.
Real-time database replication for disaster recovery
Establish a real-time replication link between two database instances for read/write splitting, read-only replicas, or disaster recovery across homogeneous or heterogeneous databases.
Real-time data migration to the cloud
Seamlessly migrate databases from an on-premises data center to a cloud database service.
Build a real-time data lake or data middle platform
Collect real-time change data from multiple operational databases into a central data lake (OSS or DLF) or data warehouse (MaxCompute or Hologres) to build a unified real-time data middle platform.
Core capabilities
Real-time database synchronization provides the following capabilities:
Core capability | Feature | Description |
Synchronize entire databases between heterogeneous data sources | - | Real-time database synchronization supports migrating data from on-premises data centers or other cloud platforms to a data warehouse or data lake such as MaxCompute, Hologres, or Kafka. For more information, see Supported data sources and synchronization solutions. |
Synchronize data in complex network environments | - | Real-time synchronization supports data from Alibaba Cloud database services, on-premises data centers, self-managed databases on ECS, and databases on other cloud platforms. Before configuration, ensure network connectivity between the resource group and the source and destination. For more information, see Network connectivity solutions. |
Synchronization scenarios | Full synchronization | Performs a one-time full data synchronization from the source to the destination tables. |
Incremental synchronization | Continuously captures streaming data from message queues or CDC logs and writes it to destination tables or specified partitions in real time. | |
Integrated full and incremental synchronization |
| |
Task configuration | Batch table synchronization | Synchronize all tables in a database, or use selection or filter rules to specify a subset of tables to synchronize. |
Automatic table creation | Configure once to process hundreds of tables from the source database. The system automatically creates table schemas in the destination without manual intervention. | |
Flexible mapping | Supports custom naming rules for destination databases and tables, as well as custom field type mappings between source and destination, flexibly adapting to the data structure model of the destination. | |
DDL change detection (supported for certain synchronization links) | When the source table schema changes (such as creating or dropping tables or columns), you can configure the synchronization task to adopt one of the following response strategies:
| |
DML rule configuration | DML message processing allows you to perform fine-grained filtering and control on the change data captured from the source ( | |
Dynamic partitioning | If the destination table is a partitioned table, dynamic partitioning is supported based on source fields or the event change time on the source side. Important Too many partitions degrade synchronization performance. If more than 1,000 partitions are added in a single day, partition creation fails and the task is terminated. | |
Task OM | Online intervention | Supports checkpoint-based resumption, which resumes execution from a specified time checkpoint after a task interruption to ensure no data loss during synchronization. Also supports re-running tasks for data backfill, anomaly repair, or logic change verification, ensuring data consistency and business continuity. |
Monitoring and alerting | Supports monitoring rules for business latency, task status, failover, and DDL notifications, and supports alerting on triggered rules. | |
Resource tuning | DataWorks Data Integration is based on serverless resource groups and provides elastic scaling at the task level. In addition, you can configure time-based elastic scaling policies to assign different resource specifications for the task at different times, such as during business peaks and off-peak hours. | |
DI Agent | AI Native | DI Agent is a vertical capability set of Data Agent for the data integration scenario. It covers six major capabilities: conversational task configuration, multimodal data processing, ChatDB data Q&A, intelligent diagnostics, periodic inspections, and IM channel integration. |
Get started
To create a real-time database synchronization task, see Create a real-time database synchronization task.
To create and manage real-time database synchronization tasks through natural language conversations, you can use the Data Agent AI Native capability. Data Agent supports all task types, including single-table offline, single-table real-time, database-wide offline, and database-wide real-time tasks. You do not need to manually fill in configuration forms. A single sentence can complete the entire process from intent understanding to task deployment. For more information, see Use Data Agent to create data integration tasks.
Supported data sources
Source data sources | Destination data sources |
MaxCompute | |
AnalyticDB for MySQL (V3.0) | |
ApsaraDB for OceanBase | |
Data Lake Formation (DLF) | |
DataHub | |
Doris | |
Elasticsearch | |
Hologres | |
Kafka | |
| LogHub |
OSS | |
OSS-HDFS | |
SelectDB | |
StarRocks | |
Lindorm |