Single-table real-time synchronization capabilities

Capability

Description

Data synchronization across various data sources

Real-time synchronization supports various data sources. You can combine different source and destination data sources to create a synchronization pipeline. For more information, see Supported data sources and sync solutions.

Data synchronization in complex network environments

Real-time synchronization supports data transfer from sources such as Alibaba Cloud databases, on-premises data centers, a self-managed database on an ECS instance, or databases outside Alibaba Cloud. Before you start, ensure that network connectivity exists between your resource group and both the source and destination. For configuration details, see Network connectivity solutions.

Synchronization scenarios

Real-time synchronization supports syncing a single table to a destination table and consolidating incremental data from sharded databases and tables into a single destination table.

• Data Integration & Data Studio (new): Configure single-table-to-single-table ETL synchronization by using a wizard. In addition to a rich set of data processing features, it also supports advanced functions like data sampling, simulation runs, and advanced parameter settings.

• Data Studio (legacy): Configure single-table-to-single-table ETL synchronization by using a drag-and-drop interface. This method supports data processing features such as data filtering, string replacement, and data masking.

Real-time synchronization task configuration

The following capabilities are supported when you configure a real-time synchronization task. You can collect single-table ETL real-time data through simple task configuration without writing code. For more information, see Configure a single-table real-time synchronization task.

Single-table real-time synchronization:

• Configuration method: Supports graphical drag-and-drop or wizard-based low-code development. No coding is required, making it easy for beginners to get started.

• Column mapping: Supports same-name mapping and same-row mapping, and allows you to customize column relationships. If a source column does not have a corresponding column in the destination table, you can specify a dynamic column handling policy to add a column, ignore the column, or report an error. Synchronization tasks also allow you to dynamically assign constant, variable, and function values to destination columns.

• Data processing: Supports performing Data Filtering, Replace String, Data Masking, and JSON Parsing on source data before outputting the processed data to the destination database.

• Code debugging: Supports sampling data from the source data source and outputting intermediate results at each data processing step. You can use Dry Run to simulate the final data output. Data output during a Dry Run is not written to the destination table, so there is no risk of affecting real data during debugging.

Real-time synchronization task O&M

Supports configuring monitoring and alerts for synchronization tasks.

• Supports resumable transfer. If a task is interrupted or data is lost due to unexpected fluctuations, you can specify an appropriate checkpoint to ensure data integrity.

• Supports configuring monitoring and alerts for business latency, failover, DDL policies, and heartbeat checks. For more information, see Configure monitoring rules for a real-time synchronization task.

• Sends alert notifications to recipients through email, SMS, phone calls, and DingTalk, so that you can promptly identify and resolve task exceptions.

• Supports alert fatigue control. To prevent a large number of alerts within a short period of time, DataWorks allows you to configure a rule to send only one alert within a specified time interval.

• Supports heartbeat detection, which automatically enables or disables heartbeat alerts when a task starts or stops. If you manually disable this feature, the current state is retained.