Batch database synchronization configuration

Actions

Description

Refresh mapping

The system automatically lists the source tables you selected, but you must refresh the mapping to apply the destination table properties.

• Select the tables to be synchronized in bulk and click Batch Refresh Mapping.

• Destination table name: The destination table name is automatically generated based on the Customize Mapping Rules for Destination Table Names rules. The default is ${SourceDatabaseName}_${TableName}. If a table with this name does not exist at the destination, the system automatically creates it.

Edit field type mapping (optional)

The system has a default mapping between source and destination field types. You can click Edit Mapping of Field Data Types in the upper-right corner to customize the mapping. After configuration, click Apply and Refresh Mapping.

When editing field type mappings, ensure that the conversion rules are correct. Otherwise, type conversion may fail, resulting in dirty data and affecting task execution.

Customize destination table name mapping (optional)

The system has a default table name generation rule: ${SourceDatabaseName}_${TableName}. You can also click the Edit button in the Customize Mapping Rules for Destination Table Names column to add a custom rule.

• Rule Name: Define a name for the rule. We recommend using a name with clear business meaning.

• Destination Table Name: You can construct the destination table name by clicking the button and combining Manually enter and Built-in Variable. Supported variables include the source data source name, source database name, and source table name.

• Edit Built-in Variables: You can apply string transformations to the built-in variables.

This feature supports the following scenarios:

1. Add prefixes or suffixes to names: Use constants to add a prefix or suffix to the source table name.

   Rule configuration	Result

2. Global string replacement: Replace the string dev_ in source table names with prd_.

   Rule configuration	Result

3. Write data from multiple tables into a single table.

   Rule configuration	Result

Customize destination database name mapping (optional)

Some destination data sources, such as Hologres, support defining mapping rules for destination databases. The configuration method is the same as for Customize destination table name mapping.

Customize destination schema name mapping (optional)

Some destination data sources, such as Hologres, support defining mapping rules for destination schemas. The configuration method is the same as for Customize destination table name mapping.

Edit destination table schema (optional)

The system automatically generates the destination table schema based on the source table schema. You do not usually need to modify it. For special requirements, you can customize it as follows:

• Add a field to a single table: Click the button in the Target Table column to add a field.

• Add fields in bulk: Select all tables to be synchronized, then at the bottom of the table, select Batch Edit > Batch Modify Table Schema.

• Renaming columns is not supported.

Value assignment for destination table fields

You must manually assign values to partition fields and any newly added fields. Standard fields are automatically mapped based on matching names in the source and destination tables. Follow these steps:

• Assign values for a single table: Click the Configuration button in the Value assignment column to assign values to the destination table fields.

• Assign values in bulk: At the bottom of the list, select Batch Edit > Value assignment to assign values to identical fields across multiple destination tables.

You can assign constants or variables. In the Value Type for Table fields and Partition fields, you can switch between the following types:

• Manual assignment: Enter a constant value or a scheduling parameter directly, such as abc, or use the ${xxx} format to enter a scheduling parameter name, such as ${bizdate}.

• Select variable: Select a system-supported variable from the drop-down list. You can view the specific meaning of each variable in the tooltip on the interface.

Both variables and scheduling parameters are automatically replaced with date-specific values when the task is scheduled to run.

Set source sharding column

In the source sharding column, you can select a field from the source table or choose Disable. When the synchronization task runs, DataWorks splits the task into multiple subtasks based on this field to enable concurrent, batched data reading.

It is recommended to use the table's primary key as the source sharding column. String, floating-point, and date types are not supported.

Currently, the source sharding column is supported only for MySQL sources.

Customize advanced parameters

Modify these parameters only if you fully understand them. Incorrect settings can cause task delays, resource contention, or data loss. You can set individual writer-side configurations and runtime configurations for subtasks.

Table type

Table Type: MaxCompute supports standard tables, PK Delta Tables, and Append Delta Tables. If the destination table status is "To be created", you can select the table type when editing the destination table schema. The type of an existing table cannot be changed.

For more information about Delta Tables, see Delta Table.