Data Integration

Limitations

You must have the Development role to create synchronization tasks in DataStudio. To learn how to grant permissions, see Add workspace members and manage their roles.

Batch synchronization tasks

• Use cases

  Batch synchronization supports two main scenarios: synchronizing a single source table to a single destination table, and synchronizing data from sharded source databases and tables to a single destination table. You can use DataWorks scheduling parameters to periodically write both incremental and full data to specific partitions in a destination table. Additionally, the data backfill feature in Operation Center lets you use a single task configuration to batch-synchronize historical data to a specific table or partition in your destination database or data warehouse.

• Supported data sources

  Data Integration supports more than 40 types of data sources, including relational databases, unstructured storage, big data storage, and message queues. You can transfer data between any structured and semi-structured data sources by defining source and destination data sources and using the Reader and Writer plug-ins in Data Integration.

• Features

  Description	References
  Data Integration uses Reader plug-ins and Writer plug-ins for batch synchronization. You can create a data source in DataWorks and use its name to specify the source and destination for the data transfer.	Supported data sources and Reader/Writer plug-ins Batch synchronization overview
  After you create a data source in DataWorks, you can configure data synchronization tasks using the codeless UI.	Configure a batch synchronization task by using the codeless UI
  You must use the code editor to configure a synchronization task in the following scenarios: The data source cannot be created directly in DataWorks. The data source does not support configuring tasks using the codeless UI. Certain plug-in parameters must be configured in the code editor.	Configure a batch synchronization task by using the code editor

Real-time synchronization tasks

Real-time synchronization lets you combine multiple data sources into star-shaped pipelines. You can configure various combinations of inputs and outputs to build flexible data synchronization links. This lets you synchronize data from a single source table to a single destination table, or from an entire source database to a destination. For details, see Supported data sources for real-time synchronization and Real-time synchronization overview.

Task scheduling configuration

Node dependencies

• Batch synchronization task
  • Ancestor nodes of a batch synchronization task: Synchronization tasks can rely on the platform's data lineage. Depending on the complexity of your workspace's business logic, you can set the task to be dependent on either the root node of the workspace or a zero load node. This setting determines whether the root node or the zero load node schedules the current node.
  • Descendant nodes of a batch synchronization task: To ensure that a downstream SQL task can automatically parse and set a dependency on this synchronization task when processing table data, add the output table of the synchronization task as the node's output. Use the format projectname.tablename.
• Descendant nodes of a real-time synchronization task
  DataWorks only supports setting dependencies on tables produced by periodically scheduled nodes. This means you cannot use data lineage to set a dependency when a downstream task needs to process data from a real-time synchronization task. As a workaround, you can set the task's ancestor to be the workspace's root node or a zero load node. This allows either the root node or the zero load node to schedule the task, depending on your business requirements.

  Note To ensure your real-time synchronization task runs as expected, configure a monitoring rule.

Node scheduling parameters