Reserve parameter reference

Updated at:

When you configure or query migration, synchronization, or tracking tasks by calling certain APIs, you must use the Reserve parameter. Reserve is a string-type parameter in JSON format. You use it to complete or view configuration details of source or destination instances. For example, you can specify the data storage format of a destination Kafka cluster or the instance ID of a Cloud Enterprise Network (CEN) instance. This topic describes the scenarios and configuration options for the Reserve parameter.

Important notes

  • You can configure common parameters based on your DTS instance type and database connection method. Then you can configure other parameters based on your source and destination database types and your actual requirements.

  • If the same parameter applies to both the source and destination databases in your DTS instance, you can configure it only once. Do not repeat the configuration.

  • If you pass a numeric value, wrap it in double quotation marks ("") to convert it to a string.

  • You can preconfigure settings in the console and preview the corresponding OpenAPI request parameters to help you fill in the request parameters. For more information, see Preview OpenAPI request parameters.

Related APIs

Common parameters

You can configure these parameters in Reserve based on your DTS instance type and database connection method.

Table 1. Migration or synchronization instances

Parameter

Required

Description

targetTableMode

Yes

How to handle an existing table in the destination database:

  • 0: Run a precheck and block the operation if an error occurs.

  • 2: Ignore errors and continue the operation.

dts.datamove.source.bps.max

No

The maximum amount of data to synchronize or migrate per second for full or incremental tasks. Unit: B (byte).

Note

Use this parameter with fullDynamicConfig or incDynamicConfig. For example, when you call an API, pass {"fullDynamicConfig": {"dts.datamove.source.bps.max":10485760},"incDynamicConfig": {"dts.datamove.source.bps.max":10485760}}.

conflict

No

The global conflict resolution policy for two-way synchronization tasks. Valid values:

  • overwrite: Overwrite conflicting records in the destination database when a conflict occurs.

  • interrupt: Stop the synchronization task and mark it as failed when a conflict occurs. You must manually fix the task.

  • ignore: Skip the current synchronization statement and continue. Keep the conflicting record in the destination database.

filterDDL

No

Whether to filter DDL operations in forward tasks for two-way synchronization. Valid values:

  • true: Do not synchronize DDL operations.

  • false: Synchronize DDL operations.

    Important

    Reverse tasks automatically filter DDL operations.

autoStartModulesAfterConfig

No

Controls whether to start modules after task configuration. Valid values:

  • none: After you configure the DTS task, do not start precheck or other modules. You must start the task manually.

  • auto: After you configure the DTS task, automatically start the precheck module and all subsequent modules.

    Note

    In the buy-then-configure scenario, the default value is auto. In the configure-then-buy scenario, this parameter does not take effect.

etlOperatorCtl

No

Whether to enable ETL features. Valid values:

  • Y: Enable ETL features.

  • No. You do not configure the ETL feature.

etlOperatorSetting

No

The ETL data processing statement. For more information, see ETL DSL syntax.

etlOperatorColumnReference

No

A field used for T+1 business. It is an ETL operator.

configKeyMap

No

ETL operator configuration information.

syncArchitecture

No

The synchronization topology. Valid values:

  • oneway: One-way synchronization.

  • bidirectional: Two-way synchronization.

dataCheckConfigure

No

Data validation configuration. For more information, see DataCheckConfigure parameter reference.

dbListCaseChangeMode

No

The case sensitivity policy for destination database object names. Valid values:

  • default: Use the default DTS policy.

  • source: Match the case of the source database.

  • dest_upper: Use uppercase, matching the default policy of the destination database.

  • dest_lower: Use lowercase, matching the default policy of the destination database.

maxRetryTime

No

The retry time in seconds after the source or destination database fails to connect. This failure is expected, such as due to network issues or lock waits. Valid values: integers from 600 to 86400. Default: 43200 seconds (720 minutes). We recommend setting it to at least 1800 seconds (30 minutes).

retry.blind.seconds

No

The retry time in seconds after unexpected failures occur in the source or destination database, such as DDL or DML execution errors. Valid values: integers from 60 to 86340. Default: 600 seconds (10 minutes). We recommend setting it to at least 600 seconds (10 minutes).

Important
  • The value of retry.blind.seconds must be less than that of maxRetryTime.

  • If multiple DTS instances connect to the same database, the shortest retry time among them takes effect.

modulesExtraConfiguration

No

Module extension configuration items. For parameter details, see Parameters reference.

Format: JSON array

Example:

[
   {
   "module": "07",
   "name": "selectdb.reservoir.group.by.target.schema",
   "value": true
   }
]

Table 2. Subscription instances

Parameter

Required

Description

vpcId

Yes

The virtual private cloud (VPC) ID of the tracking instance.

vswitchId

Yes

The ID of the virtual switch for the subscribed instance.

startTime

No

The start timestamp for data tracking. Unit: seconds.

endTime

No

The end timestamp of the data subscription, in seconds.

Table 3. Database instances connected using Cloud Enterprise Network (CEN)

Parameter

Required

Description

srcInstanceId

No

The CEN instance ID of the source instance. Example:

{
   "srcInstanceId": "cen-9kqshqum*******"  }
Note

Configure this parameter only when the source database instance uses CEN for connectivity.

destInstanceId

No

The CEN instance ID of the destination instance. Example:

{
   "destInstanceId": "cen-9kqshqum*******"  }
Note

Configure this parameter only when the destination database instance uses CEN for connectivity.

Source database parameters

You can configure these parameters in Reserve based on your source database type.

Table 4. Source database type: MySQL (including ApsaraDB RDS for MySQL and self-managed MySQL)

Parameter

Condition

Description

privilegeMigration

When both the source and destination databases are ApsaraDB RDS for MySQL.

Whether to migrate accounts. Valid values:

  • true: Migrate accounts.

  • false (default): Do not migrate accounts.

privilegeDbList

Account migration information. To determine the correct format, use Preview OpenAPI request parameters. Example:

[{"user":"user1","host":"%","oldHost":"%","is_migrate":true},{"user":"user2","host":"%","oldHost":"%","is_migrate":true}]

definer

Whether to retain the original Definer of database objects. Valid values: true or false.

amp.increment.generator.logmnr.mysql.heartbeat.mode

When the source database is a self-managed MySQL instance.

Whether to remove heartbeat table SQL statements from forward and reverse tasks. Valid values:

  • none: Do not write heartbeat SQL information to the source database.

  • N: Write heartbeat SQL information to the source database.

whitelist.dms.online.ddl.enable

When the target database type is MySQL (including ApsaraDB RDS for MySQL and self-managed MySQL), PolarDB for MySQL, AnalyticDB for MySQL, or AnalyticDB for PostgreSQL, and it is a synchronization or migration instance.

These six parameters work together to control whether to replicate temporary tables created during Online DDL changes in the source table to the destination database.

  • Replicate temporary table data generated by Online DDL changes in the source table:

    {
       "whitelist.dms.online.ddl.enable": "true",
         "sqlparser.dms.original.ddl": "false",
         "whitelist.ghost.online.ddl.enable": "true",
         "sqlparser.ghost.original.ddl": "false"
    }
  • Do not replicate temporary table data generated by Online DDL changes in the source table. Instead, synchronize only the original DDL statements executed by DMS in the source database:

    {
       "whitelist.dms.online.ddl.enable": "false",
         "sqlparser.dms.original.ddl": "true",
         "whitelist.ghost.online.ddl.enable": "false",
         "sqlparser.ghost.original.ddl": "false"
    }
  • Do not replicate temporary table data generated by Online DDL changes in the source table. Instead, synchronize only the original DDL statements executed by gh-ost in the source database:

    {
       "whitelist.dms.online.ddl.enable": "false",
         "sqlparser.dms.original.ddl": "false",
         "whitelist.ghost.online.ddl.enable": "false",
         "sqlparser.ghost.original.ddl": "true",
         "online.ddl.shadow.table.rule": "^_(.+)_(?:gho|new)$",
         "online.ddl.trash.table.rule": "^_(.+)_(?:ghc|del|old)$"
    }
    Note

    You can use the default regular expressions or customize them for gh-ost shadow tables (online.ddl.shadow.table.rule) and trash tables (online.ddl.trash.table.rule).

sqlparser.dms.original.ddl

whitelist.ghost.online.ddl.enable

sqlparser.ghost.original.ddl

online.ddl.shadow.table.rule

online.ddl.trash.table.rule

isAnalyzer

When both the source and destination databases are MySQL (including ApsaraDB RDS for MySQL and self-managed MySQL).

Whether to enable migration evaluation to check if the source and destination database structures meet requirements. Valid values: true or false.

srcSSL

When the connection method is a cloud instance or a self-managed database on ECS.

The connection method for the source database. Valid values:

  • 0: Non-encrypted connection.

  • 1: SSL secure connection.

Table 5. Source database type: PolarDB for MySQL

Parameter

Configuration requirements

Description

amp.increment.generator.logmnr.mysql.heartbeat.mode

Required

Whether to remove heartbeat table SQL statements from forward and reverse tasks. Valid values:

  • none: Do not write heartbeat SQL information to the source database.

  • N: Write heartbeat SQL information to the source database.

whitelist.dms.online.ddl.enable

When the target database type is MySQL (including ApsaraDB RDS for MySQL and self-managed MySQL), PolarDB for MySQL, AnalyticDB for MySQL, or AnalyticDB for PostgreSQL, and it is a synchronization or migration instance.

These six parameters work together to control whether to replicate temporary tables created during Online DDL changes in the source table to the destination database.

  • Replicate temporary table data generated by Online DDL changes in the source table:

    {
       "whitelist.dms.online.ddl.enable": "true",
         "sqlparser.dms.original.ddl": "false",
         "whitelist.ghost.online.ddl.enable": "true",
         "sqlparser.ghost.original.ddl": "false"
    }
  • Do not replicate temporary table data generated by Online DDL changes in the source table. Instead, synchronize only the original DDL statements executed by DMS in the source database:

    {
       "whitelist.dms.online.ddl.enable": "false",
         "sqlparser.dms.original.ddl": "true",
         "whitelist.ghost.online.ddl.enable": "false",
         "sqlparser.ghost.original.ddl": "false"
    }
  • Do not replicate temporary table data generated by Online DDL changes in the source table. Instead, synchronize only the original DDL statements executed by gh-ost in the source database:

    {
       "whitelist.dms.online.ddl.enable": "false",
         "sqlparser.dms.original.ddl": "false",
         "whitelist.ghost.online.ddl.enable": "false",
         "sqlparser.ghost.original.ddl": "true",
         "online.ddl.shadow.table.rule": "^_(.+)_(?:gho|new)$",
         "online.ddl.trash.table.rule": "^_(.+)_(?:ghc|del|old)$"
    }
    Note

    You can use the default regular expressions or customize them for gh-ost shadow tables (online.ddl.shadow.table.rule) and trash tables (online.ddl.trash.table.rule).

sqlparser.dms.original.ddl

whitelist.ghost.online.ddl.enable

sqlparser.ghost.original.ddl

online.ddl.shadow.table.rule

online.ddl.trash.table.rule

Table 6. Source database type: ApsaraDB RDS for MariaDB

Parameter

Prerequisites

Description

srcSSL

When the connection method is a cloud instance or a self-managed database on ECS.

The connection method for the source database. Valid values:

  • 0: Non-encrypted connection.

  • 1: SSL secure connection.

Table 7. Source database type: Oracle

Parameter

Condition

Description

isTargetDbCaseSensitive

When the destination database is AnalyticDB for PostgreSQL.

Whether to add quotes around target objects. Valid values: true or false.

isNeedAddRowId

When the destination database is AnalyticDB for PostgreSQL, and the objects to synchronize or migrate include tables without primary keys.

Whether to set ROWID as the primary key and distribution key for all tables without primary keys. Valid values: true or false.

srcOracleType

Required

The type of the Oracle instance. Valid values:

  • sid: Non-RAC instance.

  • serviceName: RAC or pluggable database (PDB) instance.

source.column.encoding

When you need to specify the encoding used by your application to write data.

The encoding used by your application to write data. Supported encodings:

  • default (default)

  • GB 2312

  • GBK

  • GB 18030

  • UTF-8

  • UTF-16

  • UTF-32

Table 8. Source database type: SQL Server (including ApsaraDB RDS for SQL Server and self-managed SQL Server)

Parameter

Prerequisites

Description

isTargetDbCaseSensitive

When the destination database is AnalyticDB for PostgreSQL.

Whether to add quotes around target objects. Valid values: true or false.

source.extractor.type

When the destination database is not DataHub and the task includes incremental operations.

The mode for incremental synchronization or migration of SQL Server. Valid values:

  • cdc: Use log parsing for non-heap tables and Change Data Capture (CDC) for heap tables.

  • log: Parse logs from the source database for incremental synchronization or migration.

src.sqlserver.schema.mapper.mode

When the target database type is MySQL (including ApsaraDB RDS for MySQL and self-managed MySQL), PolarDB for MySQL, or AnalyticDB for MySQL.

The schema mapping mode between the source and destination databases. Valid values:

  • schema.table: Use the Schema name.Table name from the source database as the name of the destination table.

  • without.schema: Use the table name from the source database as the name of the destination table.

    Warning

    If tables with the same name exist in multiple schemas in the source database, data inconsistency or task failure may occur.

Table 9. Source database type: Tair/Redis

Note

This includes Alibaba Cloud Tair (compatible with Redis) and self-managed Redis.

Parameter

Configuration Conditions

Description

srcKvStoreMode

When the database instance connection method is not Alibaba Cloud Instance.

The instance mode of the self-managed Redis source database. Valid values:

  • single: Basic Edition.

  • cluster: Cluster Edition.

any.sink.redis.expire.extension.seconds

Required

The additional time-to-live (TTL), in seconds, added to keys when they are migrated from the source to the destination database. To ensure data consistency, if you use any of the following commands, we recommend setting this value to at least 600 seconds.

EXPIRE key seconds
PEXPIRE key milliseconds
EXPIREAT key timestamp
PEXPIREAT key timestampMs

any.source.redis.use.slave.node

When srcKvStoreMode is cluster.

When the self-managed Redis source database is in cluster mode, choose whether to pull data from the master or slave node. Valid values:

  • true: Pull data from the slave node.

  • false (default): Pull data from the master node.

Table 10. Source database type: MongoDB (including ApsaraDB for MongoDB and self-managed MongoDB)

Parameter

Configuration Requirements

Description

srcEngineArchType

Required

The architecture type of the source MongoDB database. Valid values:

  • 0 : Standalone instance.

  • 1: ReplicaSet.

  • 2: Sharded cluster.

sourceShardEndpointUsername

When srcEngineArchType is 2.

The shard username for the source MongoDB database.

sourceShardEndpointPassword

The shard password for the source MongoDB database.

Table 11. Source database type: PolarDB-X 2.0

Parameter

Configuration Requirements

Description

amp.increment.generator.logmnr.mysql.heartbeat.mode

Required

Whether to remove heartbeat table SQL statements from forward and reverse tasks. Valid values:

  • none: Do not write heartbeat SQL information to the source database.

  • N: Write heartbeat SQL information to the source database.

Table 12. Source database type: PolarDB for PostgreSQL (Compatible with Oracle)

Parameter

Configuration Conditions

Description

srcHostPortCtl

When the connection method is a public IP address.

Whether to select multi-source data for PolarDB for PostgreSQL (Compatible with Oracle). Valid values:

  • single: Single Data Source.

  • multiple: Multiple Data Sources.

srcHostPorts

When srcHostPortCtl is multiple.

The IP addresses and port numbers of the source PolarDB for PostgreSQL (Compatible with Oracle) nodes. Separate multiple IP:Port entries with English commas.

Table 13. Source database type: TiDB

Parameter

Prerequisites

Description

amp.increment.generator.logmnr.mysql.heartbeat.mode

Required

Whether to remove heartbeat table SQL statements from forward and reverse tasks. Valid values:

  • none: Do not write heartbeat SQL information to the source database.

  • N: Write heartbeat SQL information to the source database.

isIncMigration

Required

Whether to perform incremental migration. Valid values: yes or no.

Important

Synchronization tasks support only yes.

srcKafka

When isIncMigration is yes.

Kafka information for the TiDB downstream.

taskType

The Kafka cluster type. Select based on where Kafka is deployed. Valid values:

  • EXPRESS: Leased line, VPN Gateway, or Smart Access Gateway.

  • ECS: Self-managed database on ECS.

bisId

  • When taskType is ECS, this is the ECS instance ID.

  • When taskType is EXPRESS, this is the VPC ID connected to the source database.

port

The service port of the Kafka cluster.

user

The Kafka cluster account. Leave blank if Kafka authentication is disabled.

passwd

The Kafka cluster password. Leave blank if Kafka authentication is disabled.

version

The version of the Kafka cluster.

ssl

The connection method for the Kafka cluster. Valid values:

  • 0: Non-encrypted connection.

  • 3: Encrypted connection using SCRAM-SHA-256.

topic

The topic for the migration or synchronization objects.

host

When taskType is EXPRESS.

The IP address of the Kafka cluster.

vpcId

When taskType is ECS.

The VPC of the ECS instance.

Destination database parameters

You can configure these parameters in Reserve based on your destination database type.

Table 14. Destination database type: MySQL (including ApsaraDB RDS for MySQL and self-managed MySQL)

Parameter

Condition

Description

privilegeMigration

When both the source and destination databases are ApsaraDB RDS for MySQL. For more information, see Source database type: MySQL (including ApsaraDB RDS for MySQL and self-managed MySQL).

Whether to migrate accounts.

privilegeDbList

Account migration information.

definer

Whether to retain the original Definer of database objects.

whitelist.dms.online.ddl.enable

When the source database type is MySQL (including RDS MySQL and self-managed MySQL) or PolarDB for MySQL, and the instance is a synchronization or migration instance. For more information, see Source database parameter settings.

These six parameters work together to control whether to replicate temporary tables created during Online DDL changes in the source table to the destination database.

sqlparser.dms.original.ddl

whitelist.ghost.online.ddl.enable

sqlparser.ghost.original.ddl

online.ddl.shadow.table.rule

online.ddl.trash.table.rule

isAnalyzer

When both the source and destination databases are MySQL (including ApsaraDB RDS for MySQL and self-managed MySQL).

Whether to enable migration evaluation to check if the source and destination database structures meet requirements. Valid values: true or false.

triggerMode

Required

The trigger migration method for the source database. Valid values:

  • manual: Manual migration.

  • auto: Automatic migration.

destSSL

When the connection method is a cloud instance or a self-managed database on ECS.

The connection method for the destination database. Valid values:

  • 0: Non-encrypted connection.

  • 1: SSL secure connection.

src.sqlserver.schema.mapper.mode

When the source database is SQL Server (including ApsaraDB RDS for SQL Server and self-managed SQL Server).

The schema mapping mode between the source and destination databases. For more information, see Source database type: SQL Server (including ApsaraDB RDS for SQL Server and self-managed SQL Server).

Table 15. Destination database type: PolarDB for MySQL

Parameter

Prerequisites

Description

whitelist.dms.online.ddl.enable

When the source database is MySQL (including ApsaraDB RDS for MySQL and self-managed MySQL) or PolarDB for MySQL, and the task is a synchronization or migration instance. For more information, see Source database parameters.

These six parameters work together to control whether to replicate temporary tables created during Online DDL changes in the source table to the destination database.

sqlparser.dms.original.ddl

whitelist.ghost.online.ddl.enable

sqlparser.ghost.original.ddl

online.ddl.shadow.table.rule

online.ddl.trash.table.rule

anySinkTableEngineType

Required

The engine type of the PolarDB for MySQL instance. Valid values:

  • innodb: Default storage engine.

  • xengine: On-Line Transaction Processing (OLTP) database storage engine.

triggerMode

Required

The trigger migration method for the source database. Valid values:

  • manual: Manual migration.

  • auto: Automatic migration.

src.sqlserver.schema.mapper.mode

When the source database is SQL Server (including ApsaraDB RDS for SQL Server and self-managed SQL Server).

The schema mapping mode between the source and destination databases. For more information, see Source database type: SQL Server (including ApsaraDB RDS for SQL Server and self-managed SQL Server).

Table 16. Destination database type: AnalyticDB for MySQL

Parameter

Condition

Description

whitelist.dms.online.ddl.enable

When the source database is MySQL (including ApsaraDB RDS for MySQL and self-managed MySQL) or PolarDB for MySQL, and the task is a synchronization or migration instance. For more information, see Source database parameters.

These six parameters work together to control whether to replicate temporary tables created during Online DDL changes in the source table to the destination database.

sqlparser.dms.original.ddl

whitelist.ghost.online.ddl.enable

sqlparser.ghost.original.ddl

online.ddl.shadow.table.rule

online.ddl.trash.table.rule

triggerMode

Required

The trigger migration method for the source database. Valid values:

  • manual: Manual migration.

  • auto: Automatic migration.

src.sqlserver.schema.mapper.mode

When the source database is SQL Server (including ApsaraDB RDS for SQL Server and self-managed SQL Server).

The schema mapping mode between the source and destination databases. For more information, see Source database type: SQL Server (including ApsaraDB RDS for SQL Server and self-managed SQL Server).

traceDatasource

Required

Whether to enable multi-table merge. Valid values: true or false.

tagColumnValue

When you set a custom tag column.

Whether to customize the __dts_data_source tag column. Valid values:

  • tagColumnValue: Customize the tag column.

    Important

    You must also configure the tag column value in the DbList parameter. For more information, see Migration, synchronization, or tracking object description.

  • notTagColumnValue: Do not customize the tag column.

    Important

    Custom tag columns are supported only in the buy-then-configure scenario.

adsSqlType

When you need to select incremental synchronization or migration SQL operations at the instance level.

The SQL operations for incremental synchronization or migration at the instance level. Separate multiple SQL operations with English commas. Valid values:

  • insert

  • update

  • delete

  • alterTable

  • truncateTable

  • createTable

  • dropTable

Table 17. Destination database type: AnalyticDB for PostgreSQL

Parameter

Prerequisites

Description

whitelist.dms.online.ddl.enable

When the source database is MySQL (including ApsaraDB RDS for MySQL and self-managed MySQL) or PolarDB for MySQL, and the task is a synchronization or migration instance. For more information, see Source database parameters.

These six parameters work together to control whether to replicate temporary tables created during Online DDL changes in the source table to the destination database.

sqlparser.dms.original.ddl

whitelist.ghost.online.ddl.enable

sqlparser.ghost.original.ddl

online.ddl.shadow.table.rule

online.ddl.trash.table.rule

isTargetDbCaseSensitive

When the source database is MySQL (including ApsaraDB RDS for MySQL and self-managed MySQL), Oracle, or SQL Server (including ApsaraDB RDS for SQL Server and self-managed SQL Server).

Whether to add quotes around target objects. Valid values: true or false.

syncOperation

When you need to select incremental synchronization or migration SQL operations at the instance level.

The SQL operations for incremental synchronization or migration at the instance level. Separate multiple SQL operations with English commas. Valid values:

  • insert

  • update

  • delete

  • alterTable

  • truncateTable

  • createTable

  • dropTable

  • createDB

  • dropDB

Table 18. Destination database type: ApsaraDB RDS for MariaDB

Parameter

Configuration Conditions

Description

triggerMode

Required

The trigger migration method for the source database. Valid values:

  • manual: Manual migration.

  • auto: Automatic migration.

destSSL

When the connection method is a cloud instance or a self-managed database on ECS.

The connection method for the destination database. Valid values:

  • 0: Non-encrypted connection.

  • 1: SSL secure connection.

Table 19. Destination database type: MongoDB (including ApsaraDB for MongoDB and self-managed MongoDB)

Parameter

Condition

Description

destEngineArchType

Required

The architecture type of the destination MongoDB database. Valid values:

  • 0 : Standalone instance.

  • 1: ReplicaSet.

  • 2: Sharded cluster.

Table 20. Destination database type: Tair/Redis

Note

This includes Alibaba Cloud Tair (compatible with Redis) and self-managed Redis.

Parameter

Prerequisites

Description

destKvStoreMode

When the database instance connection method is not Alibaba Cloud Instance.

The instance mode of the self-managed Redis destination database. Valid values:

  • single: Basic Edition.

  • cluster: Cluster Edition.

any.sink.redis.expire.extension.seconds

Required

The additional time-to-live (TTL), in seconds, added to keys when they are migrated from the source to the destination database. To ensure data consistency, if you use any of the following commands, we recommend setting this value to at least 600 seconds.

EXPIRE key seconds
PEXPIRE key milliseconds
EXPIREAT key timestamp
PEXPIREAT key timestampMs

Table 21. Destination database type: PolarDB for PostgreSQL (Compatible with Oracle)

Parameter

Configuration prerequisites

Description

destHostPortCtl

When the connection method is a public IP address.

Whether to select multi-source data for PolarDB for PostgreSQL (Compatible with Oracle). Valid values:

  • single: Single Data Source.

  • multiple: Multiple Data Sources.

destHostPorts

When destHostPortCtl is multiple.

The IP addresses and port numbers of the destination PolarDB for PostgreSQL (Compatible with Oracle) nodes. Separate multiple IP:Port entries with English commas.

Table 22. Destination database type: Oracle

Parameter

Condition

Description

destOracleType

Required

The type of the Oracle instance. Valid values:

  • sid: Non-RAC instance.

  • serviceName: RAC or pluggable database (PDB) instance.

Table 23. Destination database type: DataHub

Parameter

Configuration Conditions

Description

isUseNewAttachedColumn

Required

The naming rule for attached columns is as follows:

  • Use the new naming rule: Set isUseNewAttachedColumn to true.

  • Use the old naming rule: Set isUseNewAttachedColumn to false.

Table 24. Destination database type: MaxCompute

Parameter

Condition

Description

isUseNewAttachedColumn

Required

The naming rule for attached columns is as follows:

  • Use the new naming rule: Set isUseNewAttachedColumn to true.

  • Use the old naming rule: Set isUseNewAttachedColumn to false.

partition

Required

The partition name for the incremental log table. Valid values:

  • When isUseNewAttachedColumn is true:

    • modifytime_year

    • modifytime_month

    • modifytime_day

    • modifytime_hour

    • modifytime_minute

  • When isUseNewAttachedColumn is false:

    • new_dts_sync_modifytime_year

    • new_dts_sync_modifytime_month

    • new_dts_sync_modifytime_day

    • new_dts_sync_modifytime_hour

    • new_dts_sync_modifytime_minute

Table 25. Destination database type: Elasticsearch

Parameter

Prerequisites

Description

indexMapping

Required

The index name created in the destination Elasticsearch instance. Valid values:

  • tb: The index name matches the table name.

  • db_tb: The index name is formed by concatenating the database name, an underscore (_), and the table name.

Table 26. Destination database type: Kafka

Parameter

Prerequisites

Description

destTopic

Required

The topic in the destination Kafka cluster for the migration or synchronization objects.

destVersion

Required

The version of the destination Kafka cluster. Valid values: 1.0, 0.9, or 0.10.

Note

If the Kafka cluster version is 1.0 or later, enter 1.0.

destSSL

Required

The connection method for the destination Kafka cluster. Valid values:

  • 0: Non-encrypted connection.

  • 3: Encrypted connection using SCRAM-SHA-256.

sink.kafka.ddl.topic

When you need to specify a topic to store DDL information.

The topic used to store DDL information. If you do not specify one, DDL information is stored in the topic specified in destTopic by default.

kafkaRecordFormat

Required

The storage format for data delivered to the destination Kafka cluster. Valid values:

  • canal_json: Use Canal to parse database incremental logs and deliver incremental data to the destination Kafka cluster. The data storage format is Canal JSON.

  • dts_avro: A data serialization format that converts data structures or objects into formats suitable for storage or transmission.

  • shareplex_json: Use the data replication software Shareplex to read data from the source database and write it to the destination Kafka cluster. The data storage format is Shareplex JSON.

  • debezium: A tool that captures data changes and supports streaming real-time data updates from the source database to the destination Kafka cluster.

Note

For more information about formats, see Data storage formats in message queues.

destKafkaPartitionKey

Required

The Kafka partition synchronization policy. Valid values:

  • none: Deliver all data and DDL information to Partition 0 of the destination topic.

  • database_table: Combine the database name and table name as the partition key to calculate a hash value. Then deliver data and DDL information for each table to different partitions in the destination topic.

  • columns: Use columns from the table (by default, the primary key; if no primary key exists, use the unique key) as the partition key to calculate a hash value. Then deliver different rows to different partitions in the destination topic. You can also specify one or more columns as the partition key to calculate the hash value.

Note

For more information about partition synchronization policies, see Kafka partition synchronization policy reference.

destSchemaRegistry

Required

Whether to use Kafka Schema Registry. Valid values: yes or no.

destKafkaSchemaRegistryUrl

When destSchemaRegistry is true.

The URL or IP address where the Avro schema is registered in Kafka Schema Registry.

Table 27. Destination database type: OSS for lakehouse data integration tasks

Parameter

Prerequisites

Description

fusionMetastoreMod

Required

The storage location for the destination OSS metadata (data catalog). Valid values:

  • dms: Store in DMS.

  • adb: Store in an AnalyticDB for MySQL 3.0 cluster.

  • none: Do not store metadata.

fusionOssFilePath

Required

The directory in OSS where data is stored.

fusionOssFileFormat

Required

The format of integrated data in the destination OSS. Valid values:

  • DELTA: Delta format.

  • PARQUET: Parquet format.

  • HUDI: Hudi format.

fusionAdbCrossMdsDbClusterId

When fusionMetastoreMod is adb.

The ID of the AnalyticDB for MySQL 3.0 cluster that stores the destination OSS metadata.

fusionAdbCrossMdsRamId

When fusionMetastoreMod is adb.

The Alibaba Cloud account (primary account) or RAM user (sub-account) ID for the AnalyticDB for MySQL 3.0 cluster that stores the destination OSS metadata.

Note

You must have write permissions on the databases in this AnalyticDB for MySQL 3.0 cluster.

srcConnArgs

When the source database is MySQL or SQL Server.

The connection parameters for the source database in JSON format. Example:

"srcConnArgs": {
 "rewriteBatchedStatements": "true",
 "tinyInt1isBit": "false",
 "zeroDateTimeBehavior": "convertToNull",
 "yearIsDateType": "false"
 }

sparkExtraArgs

When you configure Spark task parameters.

The Spark task parameters in JSON format. Example:

 "sparkExtraArgs": {
 "spark.executor.extraJavaOptions": "-verbose:class",
 "spark.driver.extraJavaOptions": "-verbose:class"
 }