Overview of synchronization scenarios for a PolarDB for MySQL source instance
Review the notes and limits for your synchronization scenario:
Note By default, DTS disables foreign key constraints when it synchronizes data to a destination database. Therefore, operations such as cascade and delete in the source database are not synchronized to the following types of destination databases:
Synchronization between PolarDB for MySQL clusters
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not perform Data Definition Language (DDL) operations that change the schema of databases or tables. Otherwise, the data synchronization task fails.
Note During the full synchronization phase, DTS queries the source database, which acquires metadata locks. This may block DDL operations on the source database.
|
Other limits | You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Synchronization of INDEX and PARTITION is not supported. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If you perform online DDL operations that use temporary tables on the source database, such as merging multiple tables, data loss may occur in the destination database or the synchronization task may fail. If a primary key or unique key conflict occurs while the synchronization task is running: If the table schemas are the same and a record in the destination database has the same primary key or unique key value as a record in the source database: During full data synchronization, DTS retains the record in the destination cluster. The corresponding record from the source database is not synchronized. During incremental data synchronization, the record from the source database overwrites the record in the destination database.
If the table schemas are different, initial data synchronization may fail, data from only some columns may be synchronized, or the entire synchronization may fail. Proceed with caution.
DTS does not support synchronizing parsers defined by using comment syntax. If the data to be synchronized contains content that requires 4-byte storage, such as uncommon characters or emojis, the destination database and tables must use the utf8mb4 charset.
Note If you use DTS for schema synchronization, you must set the instance-level parameter character_set_server to utf8mb4 in the destination database. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If a DDL operation fails to be written to the destination database, the DTS task continues to run. You need to check the task logs for the failed DDL operation. For more information about how to view task logs, see View task logs. To synchronize accounts from the source database, you must also meet the prerequisites and understand the related considerations. For more information, see Migrate database accounts. If the task fails, DTS technical support will attempt to recover it within 8 hours. During the recovery process, operations such as restarting the task or adjusting its parameters may be performed.
Note When parameters are adjusted, only DTS task parameters are modified. Database parameters remain unchanged.The parameters that may be modified include but are not limited to those described in Modify instance parameters.
|
Other notes | Two-way synchronization between PolarDB for MySQL clusters: DTS currently supports two-way synchronization between only two PolarDB for MySQL clusters. Two-way synchronization among multiple PolarDB for MySQL clusters is not supported. DDL syntax synchronization direction limit. To ensure the stability and data consistency of the two-way synchronization link, only forward DDL synchronization is supported. Reverse DDL synchronization is not supported. When a two-way synchronization task is running, DTS creates a database named DTS in the destination databases of the forward and reverse tasks to prevent data loop synchronization. Do not modify this database while the task is running, and make sure that the database account used for the task has read and write permissions on this database. If a two-way synchronization instance in the China (Chengdu) or China (Shanghai) region includes a full data synchronization task, DTS creates a full data verification task for that instance by default. For this verification task, Full Data Verification is set to Verify based on the number of table rows. If you have already configured a full data verification task, your configuration takes precedence. A two-way synchronization instance includes a forward task and a reverse task. When you configure or reset a two-way synchronization instance, if the destination object of one task is the source object of the other task: Only one task can synchronize full and incremental data. The other task can only synchronize incremental data. The source data of the current task can only be synchronized to the destination of the current task. The synchronized data will not be used as the source data for the other task.
DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset.
|
Synchronization from PolarDB for MySQL to ApsaraDB RDS for MySQL or self-managed MySQL
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not perform Data Definition Language (DDL) operations that change the schema of databases or tables. Otherwise, the data synchronization task fails.
Note During the full synchronization phase, DTS queries the source database, which acquires metadata locks. This may block DDL operations on the source database.
|
Other limits | You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Synchronization of INDEX and PARTITION is not supported. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If you perform online DDL operations that use temporary tables on the source database, such as merging multiple tables, data loss may occur in the destination database or the synchronization task may fail. DTS does not support synchronizing parsers defined by using comment syntax. If a primary key or unique key conflict occurs while the synchronization task is running: If the table schemas are the same and a record in the destination database has the same primary key or unique key value as a record in the source database: During full data synchronization, DTS retains the record in the destination cluster. The corresponding record from the source database is not synchronized. During incremental data synchronization, the record from the source database overwrites the record in the destination database.
If the table schemas are different, initial data synchronization may fail, data from only some columns may be synchronized, or the entire synchronization may fail. Proceed with caution.
If the data to be synchronized contains content that requires 4-byte storage, such as uncommon characters or emojis, the destination database and tables must use the utf8mb4 charset.
Note If you use DTS for schema synchronization, you must set the instance-level parameter character_set_server to utf8mb4 in the destination database. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If a DDL operation fails to be written to the destination database, the DTS task continues to run. You need to check the task logs for the failed DDL operation. For more information about how to view task logs, see View task logs. If source table columns that differ only in case are synchronized to the same destination MySQL table, the synchronization result may not be as expected. This is because MySQL column names are case-insensitive. After the data synchronization is complete (the Status of the instance is Completed), we recommend that you run analyze table <table_name> to confirm that all data is written to the destination table. For example, if an HA switchover is triggered in the destination MySQL database, data might be written only to memory, which can result in data loss. A two-way synchronization instance includes a forward task and a reverse task. When you configure or reset a two-way synchronization instance, if the destination object of one task is the source object of the other task: Only one task can synchronize full and incremental data. The other task can only synchronize incremental data. The source data of the current task can only be synchronized to the destination of the current task. The synchronized data will not be used as the source data for the other task.
If the task fails, DTS technical support will attempt to recover it within 8 hours. During the recovery process, operations such as restarting the task or adjusting its parameters may be performed.
Note When parameters are adjusted, only DTS task parameters are modified. Database parameters remain unchanged.The parameters that may be modified include but are not limited to those described in Modify instance parameters.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to PolarDB-X 1.0
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During full data synchronization, do not perform DDL operations to change the schemas of databases or tables. Otherwise, the data synchronization task fails.
Note During the full synchronization phase, DTS queries the source database. This generates metadata locks, which may block the execution of DDL operations on the source database.
|
Other limits | Synchronization object requirements: Synchronization of data of the BIT, VARBIT, GEOMETRY, ARRAY, UUID, TSQUERY, TSVECTOR, and TXID_SNAPSHOT types is not supported. Synchronization of prefix indexes is not supported. If the source database has prefix indexes, the data synchronization may fail. You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster.
Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If you perform online DDL operations that use temporary tables on the source database, such as merging multiple tables, data loss may occur in the destination database or the synchronization task may fail. Initial schema synchronization is not supported. Before you configure the synchronization task, you must create the corresponding databases and tables in the destination instance. If a primary key or unique key conflict occurs while the synchronization task is running: If the table schemas are the same and a record in the destination database has the same primary key or unique key value as a record in the source database: During full data synchronization, DTS retains the record in the destination cluster. The corresponding record from the source database is not synchronized. During incremental data synchronization, the record from the source database overwrites the record in the destination database.
If the table schemas are different, initial data synchronization may fail, data from only some columns may be synchronized, or the entire synchronization may fail. Proceed with caution.
Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If the task fails, DTS technical support will attempt to recover it within 8 hours. During the recovery process, operations such as restarting the task or adjusting its parameters may be performed.
Note When parameters are adjusted, only DTS task parameters are modified. Database parameters remain unchanged.The parameters that may be modified include but are not limited to those described in Modify instance parameters.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to AnalyticDB for MySQL 2.0
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During synchronization, do not perform DDL operations to modify primary keys or add comments to tables, such as ALTER TABLE table_name COMMENT='Table comment';. Otherwise, the DDL operations fail to be executed during data synchronization. During schema synchronization and full data synchronization, do not perform Data Definition Language (DDL) operations that change the schema of databases or tables. Otherwise, the data synchronization task fails.
Note During the full synchronization phase, DTS queries the source database, which acquires metadata locks. This may block DDL operations on the source database.
|
Other limits | Synchronization of prefix indexes is not supported. If the source database has prefix indexes, the data synchronization may fail. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If you perform online DDL operations that use temporary tables on the source database, such as merging multiple tables, data loss may occur in the destination database or the synchronization task may fail. If a primary key or unique key conflict occurs while the synchronization task is running: If the table schemas are the same and a record in the destination database has the same primary key or unique key value as a record in the source database: During full data synchronization, DTS retains the record in the destination cluster. The corresponding record from the source database is not synchronized. During incremental data synchronization, the record from the source database overwrites the record in the destination database.
If the table schemas are different, initial data synchronization may fail, data from only some columns may be synchronized, or the entire synchronization may fail. Proceed with caution.
The destination database must have a custom primary key, or in the Configurations for Databases, Tables, and Columns step, configure the Primary Key Column. Otherwise, data synchronization may fail. You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. If the destination is an AnalyticDB for MySQL (2.0) cluster, the objects to be synchronized from the source instance cannot contain the reserved database names and column names of AnalyticDB for MySQL (2.0). Otherwise, the data synchronization or DDL operation synchronization fails. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If a DDL operation fails to be written to the destination database, the DTS task continues to run. You need to check the task logs for the failed DDL operation. For more information about how to view task logs, see View task logs. If the task fails, DTS technical support will attempt to recover it within 8 hours. During the recovery process, operations such as restarting the task or adjusting its parameters may be performed.
Note When parameters are adjusted, only DTS task parameters are modified. Database parameters remain unchanged.The parameters that may be modified include but are not limited to those described in Modify instance parameters.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to AnalyticDB for MySQL 3.0
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During synchronization, do not perform DDL operations to modify primary keys or add comments to tables, such as ALTER TABLE table_name COMMENT='Table comment';. Otherwise, the DDL operations fail to be executed during data synchronization. During schema synchronization and full data synchronization, do not perform Data Definition Language (DDL) operations that change the schema of databases or tables. Otherwise, the data synchronization task fails.
Note During the full synchronization phase, DTS queries the source database, which acquires metadata locks. This may block DDL operations on the source database.
|
Other limits | Synchronization of prefix indexes is not supported. If the source database has prefix indexes, the data synchronization may fail. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If you perform online DDL operations that use temporary tables on the source database, such as merging multiple tables, data loss may occur in the destination database or the synchronization task may fail. If a primary key or unique key conflict occurs while the synchronization task is running: If the table schemas are the same and a record in the destination database has the same primary key or unique key value as a record in the source database: During full data synchronization, DTS retains the record in the destination cluster. The corresponding record from the source database is not synchronized. During incremental data synchronization, the record from the source database overwrites the record in the destination database.
If the table schemas are different, initial data synchronization may fail, data from only some columns may be synchronized, or the entire synchronization may fail. Proceed with caution.
The destination database must have a custom primary key, or in the Configurations for Databases, Tables, and Columns step, configure the Primary Key Column. Otherwise, data synchronization may fail. You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. Synchronization of INDEX, PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, and FK is not supported. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Due to the limits of AnalyticDB for MySQL (3.0), if the disk space usage of a node in the AnalyticDB for MySQL (3.0) cluster exceeds 80%, the DTS task becomes abnormal and latency occurs. Estimate the required space based on the objects to be synchronized in advance to ensure that the destination cluster has sufficient storage space. If the destination AnalyticDB for MySQL 3.0 cluster is being backed up while the DTS task is running, the task fails. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If a DDL operation fails to be written to the destination database, the DTS task continues to run. You need to check the task logs for the failed DDL operation. For more information about how to view task logs, see View task logs. If the task fails, DTS technical support will attempt to recover it within 8 hours. During the recovery process, operations such as restarting the task or adjusting its parameters may be performed.
Note When parameters are adjusted, only DTS task parameters are modified. Database parameters remain unchanged.The parameters that may be modified include but are not limited to those described in Modify instance parameters.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to AnalyticDB for PostgreSQL
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During synchronization, do not perform DDL operations to modify primary keys or add comments to tables, such as ALTER TABLE table_name COMMENT='Table comment';. Otherwise, the DDL operations fail to be executed during data synchronization. If the data to be synchronized in the source database contains date data of 0000-00-00 00:00:00, the task may fail.
Note DTS converts this date data to null when synchronizing it to the destination database. You can temporarily change the source data to 0001-01-01 00:00:00 or set the corresponding field in the destination database to allow null values. During schema synchronization and full data synchronization, do not perform Data Definition Language (DDL) operations that change the schema of databases or tables. Otherwise, the data synchronization task fails.
Note During the full synchronization phase, DTS queries the source database, which acquires metadata locks. This may block DDL operations on the source database.
|
Other limits | Synchronization object requirements: Only table-level synchronization is supported. Synchronization of data of the BIT, VARBIT, GEOMETRY, ARRAY, UUID, TSQUERY, TSVECTOR, and TXID_SNAPSHOT types is not supported. Synchronization of prefix indexes is not supported. If the source database has prefix indexes, the data synchronization may fail. You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Synchronization of PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, FK, and INDEX is not supported.
Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If you perform online DDL operations that use temporary tables on the source database, such as merging multiple tables, data loss may occur in the destination database or the synchronization task may fail. If a primary key or unique key conflict occurs while the synchronization task is running: If the table schemas are the same and a record in the destination database has the same primary key or unique key value as a record in the source database: During full data synchronization, DTS retains the record in the destination cluster. The corresponding record from the source database is not synchronized. During incremental data synchronization, the record from the source database overwrites the record in the destination database.
If the table schemas are different, initial data synchronization may fail, data from only some columns may be synchronized, or the entire synchronization may fail. Proceed with caution.
If the table to be synchronized has a primary key, the primary key column of the destination table must be the same as that of the source table. If the table to be synchronized does not have a primary key, the primary key column of the destination table must be the same as the distribution key. The unique key (including the primary key column) of the destination table must contain all columns of the distribution key. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. AO tables are not supported as destination tables. If you use column mapping for non-full table synchronization or if the source and destination table schemas are inconsistent, data in the columns that are missing in the destination table will be lost. If the task fails, DTS technical support will attempt to recover it within 8 hours. During the recovery process, operations such as restarting the task or adjusting its parameters may be performed.
Note When parameters are adjusted, only DTS task parameters are modified. Database parameters remain unchanged.The parameters that may be modified include but are not limited to those described in Modify instance parameters.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to Alibaba Cloud DataHub
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not perform Data Definition Language (DDL) operations that change the schema of databases or tables. Otherwise, the data synchronization task fails.
Note During the full synchronization phase, DTS queries the source database, which acquires metadata locks. This may block DDL operations on the source database.
|
Other limits | The maximum length of a single String field in the destination DataHub topic is 2 MB. Database-level and table-level data synchronization are supported. You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If the task fails, DTS technical support will attempt to recover it within 8 hours. During the recovery process, operations such as restarting the task or adjusting its parameters may be performed.
Note When parameters are adjusted, only DTS task parameters are modified. Database parameters remain unchanged.The parameters that may be modified include but are not limited to those described in Modify instance parameters.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to Elasticsearch
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not perform Data Definition Language (DDL) operations that change the schema of databases or tables. Otherwise, the data synchronization task fails.
Note During the full synchronization phase, DTS queries the source database, which acquires metadata locks. This may block DDL operations on the source database.
|
Other limits | You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. DTS does not synchronize INDEX, PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, or FK objects. Synchronizing data to an index in the destination that has a parent-child relationship or a Join field type mapping is not supported. Otherwise, the task may become abnormal or queries in the destination may fail. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. To add columns to a table that you want to synchronize from the source database, you must first modify the mapping of the table in the Elasticsearch instance, execute the DDL operation in the source database, and then pause and start the synchronization task. The data types supported by PolarDB for MySQL and Elasticsearch do not have a one-to-one correspondence. During initial schema synchronization, DTS maps the data types based on the data types that are supported by the destination database. For more information, see Data type mappings for initial schema synchronization. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During data synchronization, if data is written to the destination database by services other than DTS, data inconsistency may occur between the source and destination databases. Elasticsearch instances of the development and test specifications are not supported. If the task fails, DTS technical support will attempt to recover it within 8 hours. During the recovery process, operations such as restarting the task or adjusting its parameters may be performed.
Note When parameters are adjusted, only DTS task parameters are modified. Database parameters remain unchanged.The parameters that may be modified include but are not limited to those described in Modify instance parameters.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command in the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to Alibaba Cloud Message Queue for Apache Kafka or self-managed Kafka
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not perform Data Definition Language (DDL) operations that change the schema of databases or tables. Otherwise, the data synchronization task fails.
Note During the full synchronization phase, DTS queries the source database, which acquires metadata locks. This may block DDL operations on the source database.
|
Other limits | You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Synchronization of INDEX, PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, and FK is not supported. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. During synchronization, if the destination Kafka cluster is scaled out or scaled in, you must restart the instance. If the task fails, DTS technical support will attempt to recover it within 8 hours. During the recovery process, operations such as restarting the task or adjusting its parameters may be performed.
Note When parameters are adjusted, only DTS task parameters are modified. Database parameters remain unchanged.The parameters that may be modified include but are not limited to those described in Modify instance parameters.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to MaxCompute
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not perform Data Definition Language (DDL) operations that change the schema of databases or tables. Otherwise, the data synchronization task fails.
Note During the full synchronization phase, DTS queries the source database, which acquires metadata locks. This may block DDL operations on the source database.
|
Other limits | You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Synchronization of INDEX, PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, and FK is not supported. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. Because MaxCompute does not support primary key constraints, if DTS retransmits data due to network issues or other reasons, duplicate records may appear in MaxCompute. If the task fails, DTS technical support will attempt to recover it within 8 hours. During the recovery process, operations such as restarting the task or adjusting its parameters may be performed.
Note When parameters are adjusted, only DTS task parameters are modified. Database parameters remain unchanged.The parameters that may be modified include but are not limited to those described in Modify instance parameters.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |
Synchronization from PolarDB for MySQL to self-managed Oracle
The following table describes the notes and limits.
Type | Description |
Source database limits | The tables to be synchronized must have a primary key or a UNIQUE constraint, and the fields must be unique. Otherwise, duplicate data may appear in the destination database. If you synchronize data at the table level and need to edit the tables, such as mapping column names, a single data synchronization task supports a maximum of 1,000 tables. If you exceed this limit, an error is reported when you submit the task. In this case, split the tables into multiple synchronization tasks or configure a task to synchronize the entire database. Binary logs: You must enable binary logging and set the loose_polar_log_bin parameter to ON. Otherwise, an error is reported during the precheck and the DTS instance fails to start. For more information about how to enable binary logging and modify parameters, see Enable binary logging and Set cluster and node parameters.
Note Enabling binary logging for a PolarDB for MySQL cluster consumes storage space and incurs fees. The binary logs of the PolarDB for MySQL cluster must be retained for at least 3 days. We recommend that you retain them for 7 days. Otherwise, the DTS task may fail because DTS cannot obtain the binary logs. In extreme cases, this may cause data inconsistency or data loss. Issues caused by a binary log retention period shorter than the required period are not covered by the DTS Service-Level Agreement (SLA).
Note For more information about how to set the retention period for binary logs of a PolarDB for MySQL cluster, see Modify the retention period.
During schema synchronization and full data synchronization, do not perform Data Definition Language (DDL) operations that change the schema of databases or tables. Otherwise, the data synchronization task fails.
Note During the full synchronization phase, DTS queries the source database, which acquires metadata locks. This may block DDL operations on the source database.
|
Other limits | You cannot synchronize data from a read-only node of the source PolarDB for MySQL cluster. You cannot synchronize OSS foreign tables from the source PolarDB for MySQL cluster. Primary/secondary failover of the database instance is not supported during initial full data synchronization. If a failover occurs, reconfigure the synchronization task promptly. If you perform online DDL operations that use temporary tables on the source database, such as merging multiple tables, data loss may occur in the destination database or the synchronization task may fail. If a primary key or unique key conflict occurs while the synchronization task is running: If the table schemas are the same and a record in the destination database has the same primary key or unique key value as a record in the source database: During full data synchronization, DTS retains the record in the destination cluster. The corresponding record from the source database is not synchronized. During incremental data synchronization, the record from the source database overwrites the record in the destination database.
If the table schemas are different, initial data synchronization may fail, data from only some columns may be synchronized, or the entire synchronization may fail. Proceed with caution.
While the DTS instance is running, disable the enabled triggers and foreign keys in the destination database. Otherwise, the DTS instance fails. Before you synchronize data, evaluate the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. Otherwise, initial full data synchronization consumes read and write resources on the source and destination databases, which may increase the database load. Initial full data synchronization runs concurrent INSERT operations, which causes fragmentation in the destination database tables. As a result, the tablespace of the destination instance is larger than that of the source instance after initial full data synchronization is complete. For table-level data synchronization, do not use tools such as pt-online-schema-change to perform online DDL operations on the synchronization objects in the source database. Otherwise, the synchronization fails. For table-level data synchronization, if no data other than the data from DTS is written to the destination database, you can use Data Management (DMS) to perform online DDL operations. For more information, see Change schemas without locking tables. During DTS synchronization, do not write data other than the data from DTS to the destination database. Otherwise, data inconsistency between the source and destination databases may occur. For example, if you use DMS to perform online DDL operations while other data is being written to the destination database, data may be lost in the destination database. If the self-managed Oracle database uses a Real Application Clusters (RAC) architecture, you cannot configure a Single Client Access Name (SCAN) IP address. You can only configure one of the virtual IP addresses (VIPs) in the connection information. After this configuration, node switching in the RAC is not supported. If the task fails, DTS technical support will attempt to recover it within 8 hours. During the recovery process, operations such as restarting the task or adjusting its parameters may be performed.
Note When parameters are adjusted, only DTS task parameters are modified. Database parameters remain unchanged.The parameters that may be modified include but are not limited to those described in Modify instance parameters.
|
Other notes | DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command on the source database to advance the binary log offset. |