Configure a real-time synchronization task (legacy)

Configure a single-table real-time synchronization task

1. Configure the input data source.

   The following input data sources and their configurations are supported for single-table data synchronization:

   • Configure MySQL input

   • Configure DataHub input

   • Configure LogHub input

   • Configure Kafka input

   • Configure PolarDB input

   1. In the Input section on the left side of the real-time synchronization task configuration page, drag the target input data source component to the panel on the right.

      

   2. Click the input component and configure its parameters in the Node Configuration panel on the right.

2. Optional: Configure data transformations.

   To transform input data into the required output format during real-time synchronization, configure data transformations.

   The following data transformations are supported for single-table data synchronization:

   • Configure data filtering: You can filter data based on rules, such as the size of a field. The task retains only data that meets the rules.

   • Configure string replacement: You can replace values in string-type fields.

   • Configure data masking: You can mask sensitive data from a single table during real-time synchronization and then store it in a specified database location.

   1. In the Conversion area on the left side of the real-time synchronization task configuration page, drag the desired data transformation component to the panel on the right. Hover over the upstream component to display its connection points. Connect the bottom point of the upstream component to the top point of the transformation component. After the components are connected, you can configure the transformation component in the Node Configuration panel.

      

   2. Click the transformation component and configure its parameters in the Node Configuration panel on the right.

3. Configure the output data source.

   The following output data sources and their configurations are supported for single-table data synchronization:

   • Configure MaxCompute output

   • Configure Hologres output

   • Configure DataHub output

   • Configure Kafka output

   • Configure Elasticsearch output

   1. In the Output area on the left side of the task configuration page, drag the target output data source component to the panel on the right and connect it to the upstream component. Configure the destination data source, table, and field mappings. If the destination table does not exist, you can click Create Table to quickly create one.

      

   2. Click the output component and configure its parameters in the Node Configuration panel on the right.

4. In the toolbar at the top of the canvas, click Save to complete the task configuration.

Configure a whole-database real-time synchronization task

1. Set the synchronization source and rules.

   1. In the Data Source area, select the Value and Data Source name of the data source that you want to synchronize.

   2. Select the tables to synchronize.

      The Select Source Table for Synchronization area displays all tables in the selected data source. In the Source Databases/Tables area, select the entire database or specific tables to synchronize, and click the icon to move them to the Selected Databases/Tables list.

      Important

      Tables without a primary key cannot be synchronized in real time.

   3. Set the mapping rules for table names.

      In this step, you can select the databases and tables from the source data source to synchronize. By default, the solution writes the source databases and data tables to a destination schema or table with the same name. If the destination schema or table does not exist, the system automatically creates it. You can also use Mapping Rules for Table Names to define the final schema or table names at the destination. This allows you to write data from multiple tables into a single table, or to uniformly update the prefix of source database or table names when writing to the destination.

      • Source and destination table name conversion rules: You can use regular expressions to convert source table names into the final destination table names.

        • Example 1: Write data from source tables with the doc_ prefix to destination tables with the pre_ prefix.

        • Example 2: Write data from multiple tables into a single destination table.

          Synchronize data from source tables named "table_01", "table_02", and "table_03" to a single table named "my_table". Configure the regular expression for table name conversion as follows: Source: table.*, Destination: my_table.

      • Rule for Destination Table Name: You can use a combination of built-in variables to generate destination table names. You can also add prefixes and suffixes to the converted destination table names. The available built-in variables are:

        • ${db_table_name_src_transed}: The table name after the conversion specified in "Source and destination table name conversion rules".

        • ${db_name_src_transed}: The destination schema name after the conversion specified in "Source and destination schema name conversion rules".

        • ${ds_name_src}: The name of the source data source.

        Example: To further process the table name converted in the previous step, use ${db_table_name_src_transed} to represent the result "my_table". You can then add a prefix and suffix to this variable, for example, pre_${db_table_name_src_transed}_post, which maps to the destination table named "pre_my_table_post".

      • Source and destination schema name conversion rules: You can use regular expressions to convert source schema names into the final destination schema names.

        Example: Replace the source database name prefix doc_ with pre_.

2. Select the destination data source and configure the destination table or topic.

   1. On the Configure Destination Table or Topic page, configure the basic information for the Destination Data Source data source, such as the write mode and partition settings. The specific configuration varies depending on the data source.

   2. Click Refresh Source and Destination Table Mapping to create the mapping between the source and destination tables.

      Use Edit additional fields to customize the destination schema, table name, and add constants or variables to the destination table. The specific configuration varies depending on the data source.

      Note

      Synchronizing a large number of tables can be slow.

3. Optional: Set table-level synchronization rules.

   This lets you define a processing strategy for insert, update, or delete operations that occur in the source table.

   Note

   The availability of DML processing policies depends on the specific synchronization solution. For the current DML support status of data sources, see Supported DML and DDL operations.

4. Set DDL message processing rules.

   Source data sources can involve many DDL operations. During real-time synchronization, you can set processing policies for different DDL messages based on your business needs. Supported DDL operations may vary by data source. For more information, see Supported DML and DDL operations. You can set DDL processing policies for each destination database type on the Data Integration > Configuration Options > Processing Policy for DDL Messages in Real-time Sync page. The following table describes the DDL message processing policies.

   DDL message type	Processing policy
   Create Table	When DataWorks receives a DDL message of the corresponding type, the processing policy is as follows: Normal: Forwards the message to the destination data source for processing. DataWorks only forwards the message; the actual handling depends on the destination data source. Ignored Object: Discards the message without sending it to the destination data source. Alert: Discards the message and records an alert in the real-time synchronization log, indicating that an execution error occurred. Error: The real-time synchronization task immediately enters an error state and stops running.
   Delete Table
   Insert Column
   Delete Column
   Rename Table
   Rename column
   Modify Column Type
   Clear Table

5. Configure runtime resources.

   • You can control the task's concurrency, which is the maximum number of threads that Data Integration uses to read from and write to databases.

   • You can control whether the synchronization task tolerates dirty data.

     • If dirty data is not allowed, the task fails if it encounters any dirty data.

     • If dirty data is allowed, the synchronization task ignores the dirty data (without writing it to the destination) and continues to run.

6. Click Complete.

For a single-table real-time integration task that is configured in Data Studio, click Migrate to Data Integration to migrate the task to the Data Integration page.

1. Double-click the single-table real-time integration task that you want to migrate to open its configuration page, and click Migrate to Data Integration to migrate the task.

   

2. In the upper-left corner, click and select All Products > Data Integration. On the Synchronization Task page, you can find the successfully migrated single-table real-time integration task in the task list.