Configure real-time full-database synchronization tasks

Operation

Description

Refresh

The system automatically lists the source tables you selected, but the specific properties of the destination tables take effect only after you refresh and confirm them.

• Select the tables to be synchronized in batch and click batch refresh mapping.

• Destination table name: The destination table name is automatically generated based on the Customize Mapping Rules for Destination Table Names rules. The default format is ${source_database_name}_${table_name}. If a table with the same name does not exist in the destination, the system automatically creates one.

Customize Mapping Rules for Destination Table Names (optional)

The system has a default table name generation rule: ${source_database_name}_${table_name}. You can also click the Edit button in the Customize Mapping Rules for Destination Table Names column to add a custom destination table name rule.

• Rule name: Define the rule name. We recommend that you specify a name with a clear business meaning.

• Destination table name: You can click the button and select Manually enter and Built-in Variable to concatenate and generate the destination table name. The supported variables include the source data source name, source database name, and source table name.

• Edit built-in variables: You can perform string transformations on built-in variables.

The following scenarios are supported:

1. Add prefixes or suffixes to table names: Add a prefix or suffix to the source table name by setting constants.

   Rule configuration	Result

2. Unified string replacement: Replace the string dev_ in the source table name with prd_.

   Rule configuration	Result

3. Write multiple tables to a single table: Set the destination table name to a constant.

   Rule configuration	Result

Both MaxCompute and Hologres support target schema name mapping customization. When you synchronize a full MySQL database to MaxCompute in real time, you can configure target schema name mapping customization rules to route synchronized data to a specified non-default schema in the destination MaxCompute project, rather than writing all data to the default schema.

Edit column type mapping (optional)

The system has default mappings between source column types and destination column types. You can click Edit Mapping of Field Data Types in the upper-right corner of the table to customize the column type mappings between the source and destination tables, and then click Apply and Refresh Mapping.

When editing column type mappings, make sure that the type conversion rules are correct. Incorrect rules may cause type conversion failures, resulting in dirty data and affecting task execution.

Edit destination table schema (optional)

The system automatically creates destination tables that do not exist based on the custom table name mapping rules, or reuses existing tables with the same name.

DataWorks automatically generates the destination table schema based on the source table schema. In most cases, no manual intervention is required. You can also modify the table schema in the following ways:

• Add columns to a single table: Click the button in the Target Table column to add columns.

• Add columns in batches: Select all tables to be synchronized, and choose Batch Edit > Destination Table Schema - Batch Modify and Add Field at the bottom of the table.

• Renaming column names is not supported.

For existing tables, you can only add columns. For new tables, you can add columns, partition columns, and configure the table type or table properties. For details, see the editable areas on the page.

Value assignment

Native columns are automatically mapped based on columns with the same name in the source and destination tables. The added columns and partition columns in the preceding steps require manual value assignment. Perform the following operations:

• Assign values for a single table: Click the Configuration button in the Value assignment column to assign values to destination table columns.

• Assign values in batches: Select Batch Edit > Value assignment at the bottom of the list to assign values in batches for columns with the same name in the destination tables.

When assigning values, you can use constants or variables. Switch the type in Value Type. The following methods are supported:

• Table columns

  • Manual assignment: Directly enter a constant value, such as abc.

  • Select a variable: Select a system-supported variable from the drop-down list. You can view the meaning of each variable in the tooltip on the page.

  • Function: You can use a function to apply a simple transformation to the destination column. For more information, see Use functions for column value transformation.

• Partition columns: Partitions can be dynamically created based on the enumeration values of source columns or the event time as partition values.

  • Manual assignment: Directly enter a constant value, such as abc.

  • Source column: Use the value of a source table column as the partition column value. The value type can be a column value or a time value.

    • Column value: The enumeration values in the source column. We recommend that you use columns with a limited number of enumeration values to prevent excessive partitions and overly scattered data.

    • Time value: If the value in the source column is a timestamp, you can process it based on different formats and specify Target format to format the partition value.

      • Time string: A string that represents a time value, such as "2018-10-23 02:13:56" or "2021/05/18". Serialize the string into a time value by specifying the time format on the source and destination sides. For the example strings above, you can use the yyyy-MM-dd HH:mm:ss and yyyy/MM/dd formats for serialization and recognition.

      • Time object: If the source value is already a time type such as Date or Datetime, select this type directly.

      • Unix timestamp (seconds): A second-level timestamp. It also supports numbers or strings in a 10-digit timestamp format, such as 1610529203 and "1610529203".

      • Unix timestamp (milliseconds): A millisecond-level timestamp. It also supports numbers or strings in a 13-digit timestamp format, such as 1610529203002 and "1610529203002".

  • Select a variable: You can use the source event change time EVENT_TIME as the partition value source. The usage is similar to that of source columns.

  • Function: You can use a function to apply a simple transformation to a source column and use the result as the partition value. For more information, see Use functions for column value transformation.

Source Split Key

You can select a column from the source table or select Not Split from the drop-down list of the source split key. During task execution, the data is split into multiple tasks based on this column to enable concurrent and batch data reads.

We recommend that you use the primary key of the table as the source split key. String, floating-point, date, and other types are not supported.

Currently, the source split key is supported only when the source is MySQL.

Skip Full Synchronization

If full synchronization has been configured in Step 3, you can individually cancel full data synchronization for a specific table. This is applicable to scenarios where full data has already been synchronized to the destination by other means.

Full condition

Apply conditional filtering to the source during the full synchronization phase. Enter only the WHERE clause here without the WHERE keyword.

Configure DML Rule

DML message processing is used to perform fine-grained filtering and control on change data captured from the source (Insert, Update, Delete) before the data is written to the destination. This rule takes effect only during the incremental synchronization phase.

Others

Table Type: MaxCompute supports regular tables, PK Delta Table, and Append Delta Table. If the destination table status is To Be Created, you can select the table type when editing the destination table structure. The type of an existing table cannot be changed.

• The full + incremental mode of real-time full-database synchronization supports only PK Delta Table as the destination table type.

• In incremental-only mode, the replay mode supports PK Delta Table, and the incremental streaming mode supports regular table types and Append Delta Table.

For more information about Delta Table, see Delta Table overview.