Batch synchronization features

Feature

Description

Full or incremental data synchronization

Batch synchronization tasks support full or incremental data synchronization by configuring Data Filtering combined with scheduling parameters. Different plugins have different configurations for incremental synchronization. For more information about incremental data synchronization, see Configure incremental data synchronization.

Field mapping

By defining field mapping rules, source data is written to the corresponding destination fields based on the specified relationships. Ensure that the field types on both sides are compatible.

• Multiple field mapping methods are available:

  • Wizard mode supports same-name mapping, same-row mapping, and custom field relationships. Data in unmapped fields is automatically ignored. Ensure that the corresponding destination fields have default values configured or allow null values to avoid write failures.

  • Script mode strictly maps fields based on the column configuration order. The number of fields on the reader side must exactly match the number on the writer side. Otherwise, the task fails at runtime.

• Synchronization tasks also support dynamic value assignment for destination fields, including constants, scheduling parameters, and built-in variables such as ${bizdate}. The relevant parameters are assigned final values during the scheduling phase.

Job rate limit

• The task concurrency control feature limits the maximum number of concurrent threads for reading from and writing to databases.

• The sync speed feature controls traffic to prevent excessive pressure on the source or destination caused by high synchronization speeds. If throttling is not enabled, the maximum transfer performance available in the current hardware environment is provided.

Distributed task execution

Data sources that support distributed execution can use task sharding to distribute synchronization tasks across multiple nodes for concurrent execution. This enables synchronization speed to scale linearly with cluster size, breaking through single-node performance bottlenecks. This mode is especially suitable for high-throughput, low-latency synchronization scenarios and efficiently utilizes idle cluster resources to significantly improve hardware utilization.

Dirty data policy

Dirty data refers to data records that fail to be written to the destination due to exceptions such as type conflicts or constraint violations. Batch synchronization supports dirty data policies that define the acceptable number of dirty data records and their impact on tasks.

• Ignore dirty data: Dirty data is automatically filtered out. Only valid data is written, and the task continues running.

• Tolerate limited dirty data: Set a threshold N. If the number of dirty data records is ≤ N, the abnormal data is discarded and the task continues. If the number exceeds N, the task fails and exits.

• No tolerance for dirty data: The task immediately fails and exits when dirty data is encountered.

Time zone

If cross-time-zone synchronization is required between the source and destination, you can configure the source time zone to perform time zone conversion.

Intelligent data processing

DataWorks supports integrating data processing capabilities during data synchronization to transform and process source data before writing it to the destination:

String replacement: The built-in string replacement feature in batch synchronization tasks allows you to perform lightweight data transformations directly during data transfer without requiring data to be landed or additional ETL steps.

AI-assisted processing: Supports integrating AI large language models during data synchronization to perform semantic analysis, sentiment analysis, and other processing on source natural language data, and writes the processed results directly to the destination table.

Data vectorization: Supports extracting and vectorizing (embedding) source data before writing it to a vector database.