Configure real-time full-database synchronization

Operation

Description

Refresh

The system automatically lists the source tables you selected, but the specific properties of the destination tables take effect only after you refresh and confirm them.

• Select the tables to synchronize in batches and click Batch Refresh Mapping.

• Destination table name: The destination table name is automatically generated based on the Customize Mapping Rules for Destination Table Names rules. The default format is ${source_database_name}_${table_name}. If a table with the same name does not exist on the destination, the system automatically creates one.

Customize Mapping Rules for Destination Table Names (optional)

The system has a default table name generation rule: ${source_database_name}_${table_name}. You can also click the Edit button in the Customize Mapping Rules for Destination Table Names column to add custom destination table name rules.

• Rule name: Define a rule name. We recommend that you use a name with a clear business meaning.

• Destination table name: You can click the button to select Manually enter and Built-in Variable to concatenate and generate the destination table name. The supported variables include source data source name, source database name, and source table name.

• Edit built-in variables: Built-in variables support string transformations on top of the original values.

The following scenarios are supported:

1. Add a prefix or suffix to a name: Add a prefix or suffix to the source table name by setting a constant.

   Rule configuration	Result

2. Unified string replacement: Replace the string dev_ in the source table name with prd_.

   Rule configuration	Result

3. Write multiple tables to a single table: Set the destination table name to a constant.

   Rule configuration	Result

Edit column type mapping (optional)

The system has default column type mappings between source and destination. You can click Edit Mapping of Field Data Types in the upper-right corner of the table to customize the column type mapping between source and destination tables, and then click Apply and Refresh Mapping.

When editing column type mappings, make sure the type conversion rules are correct. Incorrect rules may cause type conversion failures that produce dirty data and affect task execution.

Edit destination table schema (optional)

The system automatically creates destination tables that do not exist based on the custom table name mapping rules, or reuses existing tables with the same name.

DataWorks automatically generates the destination table schema based on the source table schema. Manual intervention is not required in most cases. You can also modify the table schema in the following ways:

• Add columns to a single table: Click the button in the Target Table column to add columns.

• Add columns in batches: Select all tables to synchronize, and choose Batch Edit > Destination Table Schema - Batch Modify and Add Field at the bottom of the table.

• Renaming column names is not supported.

For existing tables, you can only add columns. For new tables, you can add columns and partition columns, and set the table type or table properties. For details, see the editable areas on the interface.

Value assignment

Native columns are automatically mapped based on columns with the same name in the source and destination tables. The newly added columns and partition columns from the preceding steps must be manually assigned values. Perform the following operations:

• Assign values to a single table: Click the Configuration button in the Value assignment column to assign values to destination table columns.

• Assign values in batches: Choose Batch Edit > Value assignment at the bottom of the list to assign values in batches to the same columns in the destination table.

When assigning values, you can assign constants and variables. Switch the type in Value Type. The following methods are supported:

• Table columns

  • Manual input: Directly enter a constant value, such as abc.

  • Select a variable: Select a system-supported variable from the drop-down list. You can view the specific meaning of the variable in the tooltip on the interface.

  • Function: You can use functions to perform simple transformations on the destination columns. For more information, see Function reference.

• Partition columns: You can dynamically create partitions by using the enumeration values of a source column or the event time as the partition value.

  • Manual input: Directly enter a constant value, such as abc.

  • Source column: Use the values of a source table column as the partition column values. The value type can be a column value or a time value.

    • Column value: The enumeration values from the source column. We recommend that you use a column with a limited number of enumeration values to prevent too many partitions and overly scattered data.

    • Time value: If the values in the source column are timestamps, you can process them based on different formats and specify a Target format to format the partition values.

      • Time string: A string that represents a time, such as "2018-10-23 02:13:56" or "2021/05/18". You can serialize it into a time value by specifying the source and destination time formats. For the preceding examples, you can use the yyyy-MM-dd HH:mm:ss and yyyy/MM/dd formats for serialization.

      • Time object: If the source value is already a time type such as Date or Datetime, select this type directly.

      • Unix timestamp (seconds): A second-level timestamp. This also supports numbers or strings in the 10-digit timestamp format, such as 1610529203 or "1610529203".

      • Unix timestamp (milliseconds): A millisecond-level timestamp. This also supports numbers or strings in the 13-digit timestamp format, such as 1610529203002 or "1610529203002".

  • Select a variable: You can use the source event change time EVENT_TIME as the partition value source. The usage is similar to that of source columns.

  • Function: You can use functions to perform simple transformations on the source column before using the result as a partition value. For more information, see Function reference.

Source Split Key

You can select a column from the source table or select Not Split in the source split key drop-down list. During task execution, data is split into multiple tasks based on this column to enable concurrent, batch data reads.

We recommend that you use the table primary key as the source split key. String, floating-point, date, and other types are not supported.

Currently, the source split key is supported only when the source is MySQL.

Skip Full Synchronization

If you have configured full synchronization in Step 3, you can individually skip full data synchronization for specific tables. This is applicable to scenarios where full data has already been synchronized to the destination through other methods.

Full condition

Apply conditional filtering to the source during the full synchronization phase. You only need to write the WHERE clause here, without the WHERE keyword.

Configure DML Rule

DML message processing is used to perform fine-grained filtering and control on change data captured from the source (Insert, Update, Delete) before writing the data to the destination. This rule takes effect only during the incremental phase.

Others

Table Type: MaxCompute supports regular tables, PK Delta Table, and Append Delta Table. If the destination table status is to be created, you can select the table type when editing the destination table schema. The type cannot be changed for existing tables.

• The full + incremental mode of real-time full-database synchronization supports only the PK Delta Table table type as the destination.

• In incremental-only mode, the replay mode supports PK Delta Table, and the append mode supports regular table types and Append Delta Table.

For more information about Delta Table, see Delta Table overview.