Migrate data between PolarDB for PostgreSQL (Compatible with Oracle) clusters

Usage notes

• During a full data migration, DTS consumes read and write resources on the source and destination databases, increasing their load. If your databases have poor performance, low specifications, or high workloads (for example, if the source database has many slow SQL queries or tables without primary keys, or if deadlocks occur in the destination database), the increased load can strain your databases or even cause service interruptions. Perform the data migration during off-peak hours, such as when the CPU utilization of both databases is below 30%.

• A data migration task can migrate data from only one database at a time. To migrate data from multiple databases, you must configure a separate data migration task for each database.

• During incremental data migration, if you select a schema as the migration object and then create a new table or rebuild a table by using RENAME, run ALTER TABLE schema.table REPLICA IDENTITY FULL; before writing data to the table.
  Note Replace schema and table with the actual schema and table names.

• To ensure the accuracy of the latency display for incremental data migration, DTS adds a heartbeat table named dts_postgres_heartbeat. The table contains the following columns:

  • SLOT_NAME: The name of the logical replication slot that DTS creates in the source database. During incremental data migration, DTS retrieves the WAL of the source database through this replication slot.

  • REVICE_TIME: The time when the heartbeat record was written to the source database.

  • REVICE_LSN: The log sequence number (LSN) of the WAL in the source database at the time when the heartbeat record was written.

  • FLUSHED_LSN: The LSN of the WAL in the source database up to which DTS has read. This value reflects the progress of incremental data migration.

  • UPDATE_TIME: The time when the heartbeat record was last updated.

  • DTS_SERVICE_TIME: The time when the DTS service processes the heartbeat record. This value is used to calculate the latency of incremental data migration.

• DTS triggers automatic recovery for failed migration tasks. Before you switch your workloads to the destination database, make sure that you stop or release the migration task. Otherwise, the task may be automatically recovered and overwrite data in the destination database with data from the source database.

• If the source database has long-running transactions and the instance contains incremental migration tasks, the write-ahead log (WAL) generated before the long-running transactions are committed may accumulate and cannot be cleaned up, which may cause insufficient disk space in the source database.

Billing

Migration types

Procedure

1. Log on to the DTS console.

   Note

   If you are automatically redirected to the Data Management (DMS) console, you can click the icon in the lower-right corner and then click to return to the classic DTS console.

2. In the left-side navigation pane, click Data Migration.

3. At the top of the Migration Tasks page, select the region of the destination cluster.

4. In the upper-right corner of the page, click Create Data Migration Task.

5. Configure the source and destination database connection settings for the migration task.

   Category	Parameter	Description
   N/A	Task Name	DTS automatically generates a task name. We recommend that you specify a descriptive name that makes it easy to identify the task. Task names do not need to be unique.
   Source Database	Instance Type	Select PolarDB.
   	Instance Region	Select the region where the source PolarDB cluster resides.
   	PolarDB Instance ID	Select the ID of the source PolarDB cluster.
   	Database Name	Enter the name of the database to be migrated.
   	Database Account	Enter a privileged account of the source PolarDB cluster. For more information, see Create a database account.
   	Database Password	Enter the password of the database account. Note After you enter the source database information, you can click Test Connectivity next to Database Password to verify that the information is correct. If the information is correct, the message Passed is displayed. If the message Failed is displayed, click Diagnose next to the Failed message and adjust the source database information based on the prompts.
   Destination Database	Instance Type	Select PolarDB.
   	Instance Region	Select the region where the destination PolarDB cluster resides.
   	PolarDB Instance ID	Select the ID of the destination PolarDB cluster.
   	Database Name	Enter the name of the database to which data is migrated.
   	Database Account	Enter the database account of the destination PolarDB cluster. The account must have the Database Owner permission. Important The Database Owner is specified when the database is created.
   	Database Password	Enter the password of the database account. Note After you enter the destination database information, you can click Test Connectivity after Database Password to verify that the entered information is correct. If the information is correct, a Passed message is displayed. If a Failed message is displayed, click Diagnose after Failed and adjust the destination database information based on the prompts.

6. After the configuration is complete, click Authorize Whitelist and Proceed in the lower-right corner.

   If the source or destination database is an Alibaba Cloud database instance, such as an RDS for MySQL or ApsaraDB for MongoDB instance, DTS automatically adds the CIDR blocks of DTS servers to the IP address whitelist of the instance. If the source or destination database is a self-managed database hosted on an ECS instance, DTS automatically adds the CIDR blocks of DTS servers to the security group rules of the ECS instance. You must also ensure that the database is accessible from the ECS instance. If the self-managed database is hosted on multiple ECS instances, you must manually add the CIDR blocks of DTS servers to the security group rules of each ECS instance. If the source or destination database is a self-managed database that is deployed in an on-premises data center or provided by a third-party cloud service provider, you must manually add the CIDR blocks of DTS servers to the IP address whitelist of the database to allow access from DTS servers. For more information about the IP addresses of DTS servers, see CIDR blocks of DTS servers.

   Warning

   Adding the public CIDR blocks of DTS servers, whether automatically or manually, may introduce security risks. By using this product, you acknowledge and accept these potential risks. You are responsible for implementing basic security measures, including but not limited to using strong passwords, restricting open ports, using authentication for internal API calls, regularly reviewing and restricting unnecessary network segments, or connecting through private networks such as Express Connect, VPN Gateway, or Smart Access Gateway.

7. Select the migration types and migration objects.

   Parameter	Description
   Migration Type	To perform only full data migration, select Schema Migration and Full Data Migration. To perform migration without downtime, select Schema Migration, Full Data Migration, and Incremental Data Migration. Important If Incremental Data Migration is not selected, do not write new data to the source database during schema migration or full data migration to ensure data consistency. Do not perform DDL operations on the migration objects before schema migration and full data migration are complete. Otherwise, the migration task may fail.
   Migration Objects	In the Migration Objects section, click the objects to be migrated (schemas, tables, or columns), and then click to move them to the Selected Objects section. Important By default, the names of the migration objects remain unchanged after the migration is complete. To change the names of the migration objects in the destination PolarDB cluster, use the object name mapping feature. For more information, see Object name mapping. If you use the object name mapping feature, other objects that depend on the renamed object may fail to be migrated.
   Mapping Name Change	To change the names of the migration objects in the destination instance, use the object name mapping feature. For more information, see Object name mapping.
   Retry Time for Connection Failures	When the source or destination database cannot be connected, DTS retries the connection for 720 minutes (12 hours) by default. You can also customize the retry period. If DTS reconnects to the source and destination databases within the specified period, the migration task is automatically resumed. Otherwise, the migration task fails. Note DTS charges fees during the retry period. We recommend that you customize the retry period based on your business needs, or release the DTS instance as soon as possible after the source and destination database instances are released.

8. After you complete the configuration, click Precheck and Start in the lower-right corner of the page.

   Note

   • Before the migration task starts, DTS runs a precheck. The task can start only after it passes the precheck.

   • If the precheck fails, click the icon next to the failed item to view details.

     • Fix the issues as prompted and run the precheck again.

     • If you do not need to fix the warning items, you can select Ignore and then click Ignore Warnings and Rerun Precheck to run the precheck again.

9. After the task passes the precheck, click Next.

10. In the Confirm Settings dialog box that appears, select a Instance Class and select the Data Transmission Service (pay-as-you-go) Service Terms checkbox.

11. Click Buy and Start to begin the migration.

    • Schema migration + Full data migration

      Allow the task to complete automatically. Stopping it manually may result in incomplete data.

    • Schema migration + Full data migration + Incremental data migration

      The migration task does not stop automatically. You must stop it manually.

      Important

      Choose an appropriate time to stop the task manually, such as during off-peak hours or when you are ready to switch your business to the destination cluster.

      1. Wait until the migration task enters the Incremental Data Migration phase and the status shows Undelayed. Then, stop writing data to the source database for several minutes. During this time, the status of Incremental Data Migration may show a latency.

      2. Wait for the Incremental Data Migration status to show Undelayed again. Then, manually stop the migration task. Select the check box of the target migration task, and click Pause in the batch operation bar at the bottom of the page.

Related documentation

Workload switchover procedure