Migrate data using Data Transmission Service

Prerequisites

Before you migrate data, confirm the following:

• Data Transmission Service has the required access permissions for the cloud resources. For more information, see Grant permissions to a Data Transmission Service migration role.

• You have created the source and destination data sources. For more information, see Create a data source.

• The source and destination databases have network connectivity.

• You have created a database user for data migration in both the source MySQL database and the MySQL tenant of the destination OceanBase database, and granted the required permissions to the user. For more information, see Configure data source user permissions.

• Binary logging (binlog) is enabled. For more information, see Manage binary logs (binlog).

Procedure

1. Log on to the OceanBase Management Console and purchase a data migration task.

   For more information, see Purchase a data migration task.

2. On the Data Transmission > Data Migration page, click Configure for your new data migration task.

   

   To reference an existing task configuration, you can click Reference Configuration. For more information, see Reference a data migration task configuration.

3. On the Select Source And Target page, configure the parameters.

   Parameter	Description
   Migration Task Name	The name should consist of a combination of Chinese characters, digits, and letters. It cannot contain spaces and must not exceed 64 characters.
   Source	The name of the source database. In this topic, a MySQL database is used as the source. If you have already created a MySQL data source, select it from the drop-down list. If you have not created one, click Create Data Source in the drop-down list and create one in the dialog box that appears on the right. For more information about the parameters, see Create a MySQL data source.
   Destination	The name of the destination database. In this topic, the destination is an OceanBase database. If you have created a MySQL tenant data source for the OceanBase database, select it from the drop-down list. If not, click Create Data Source in the drop-down list and create one in the dialog box on the right. For details about the parameters, see Create an OceanBase data source. Important The destination instance does not support an OceanBase database whose Instance Type is Self-managed Database In A VPC.
   Tags (Optional)	Click the text box and select the target tag from the drop-down list. You can also click Manage Tags to create, modify, and delete tags. For more information, see Manage data migration tasks using tags.

4. Click Next. On the Select Migration Type page, set Synchronization Topology to One-way Synchronization.

   Some types of data migration tasks support One-way Synchronization and Two-way Synchronization. This topic describes the operations for one-way synchronization. For more information about two-way synchronization, see Configure a two-way synchronization task.

5. Select the migration types for the task.

   For a one-way synchronization topology, the following migration types are supported: Schema Migration, Full Migration, Incremental Synchronization, Full Data Verification, and Reverse Incremental.

   

   Migration Type	Description
   Schema Migration	After the schema migration task starts, Data Transmission Service migrates the definitions of data objects (such as tables, indexes, constraints, comments, and views) from the source database to the destination database and automatically filters out temporary tables. For schema migration from a MySQL database to an OceanBase database in MySQL-compatible mode, Data Transmission Service supports the automatic creation of databases that do not exist in the destination.
   Full Migration	After a full migration task starts, Data Transmission Service migrates the existing data in the source tables to the corresponding tables in the destination database. If you select Full Migration, we recommend that you use the ANALYZE statement to collect statistics for the MySQL database before you migrate data.
   Incremental Synchronization	After the incremental synchronization task starts, Data Transmission Service synchronizes the changed data (new, modified, or deleted) from the source database to the corresponding tables in the destination database. Incremental Synchronization includes DML Synchronization and DDL Synchronization. You can customize the configurations based on your requirements. For more information, see Customize DDL/DML. The limits of Incremental Synchronization are as follows: If you select DDL Synchronization, the data migration is at risk of interruption if the source database performs an incremental DDL operation that the data transmission process does not support. If the DDL operation is to add a column, we recommend that you set the attribute of the column to NULL. Otherwise, the data migration may be interrupted.
   Full Validation	After full migration is complete and the incremental data is synchronized to the destination and is almost caught up with the source, Data Transmission Service automatically initiates a full data validation task for the data tables configured for the source database and the destination tables. If you select Full Data Verification, we recommend that you collect statistics information for both the MySQL database and the OceanBase database in MySQL compatible mode before the full data verification starts. If you select Incremental Synchronization and do not select all DML operations in the DML Synchronization option, Data Transmission Service does not support full data verification in this scenario. Data Transmission Service supports full data validation only for tables with unique keys (tables with a primary key or a non-null unique key).
   Reverse Incremental Synchronization	After the reverse incremental synchronization task starts, it can synchronize the data changes generated in the destination database back to the source database in real time after a service switchover. Reverse incremental synchronization usually reuses the configuration of incremental synchronization. You can also customize the configuration as needed.

6. Click Next. On the Select Migration Objects page, select the migration objects for the current data migration task.

   You can use Specify Objects or Matching Rules to select migration objects.

   • Select Specify Objects. In the left pane, select the objects to migrate and click > to add them to the list on the right. You can select tables and views from one or more databases as migration objects. The Data Transmission Service supports importing objects from a text file. It also supports operations such as renaming destination objects, setting row filtering, viewing column information, and removing one or more migration objects. The steps to import objects are as follows:

     1. In the list on the right side of the selection area, click Import Object in the upper-right corner.

     2. In the dialog box, click OK.

        Important

        Importing will overwrite previous selections. Proceed with caution.

     3. In the Import Migration Objects dialog box, import the objects.

        You can import a CSV file to perform operations such as renaming databases and tables and setting row filtering conditions. For more information, see Download and import migration object configurations.

     4. Click Check Validity.

        After you import the migration objects, you must check their validity. Column field mapping is not currently supported.

     5. After the validation is successful, click OK.

   • Select Matching Rule. For more information, see Configure and modify matching rules.

7. Click Next. On the Migration Options page, configure the parameters.

   • Full Migration

     On the Select Migration Type page, the following parameters appear only when you select One-way Synchronization > Full Migration.

     

     Parameter	Description
     Read Concurrency	This parameter specifies the number of concurrent threads for reading data from the source during full migration. The maximum value is 512. A high degree of concurrency may put excessive pressure on the source and affect your services.
     Write Concurrency	This parameter specifies the number of concurrent threads for writing data to the destination during full migration. The maximum value is 512. A high degree of concurrency may put excessive pressure on the destination and affect your services.
     Full Migration Rate Limit	You can decide whether to enable the full migration rate limit as needed. If you enable it, set RPS (the maximum number of data rows that can be migrated to the destination per second during full migration) and BPS (the maximum volume of data that can be migrated to the destination per second during full migration). Note The RPS and BPS settings here only serve as a rate-limiting capability. The actual performance of full migration is affected by factors such as the source, destination, and instance specifications.
     Processing Policy for Existing Records in Destination Table Objects	The processing policy includes Ignore and Stop Migration: Select Ignore: If a destination table already contains data and a conflict occurs between the data being written and the existing data, Data Transmission Service logs the conflicting data and keeps the existing data unchanged. Important If you select Ignore, full data verification uses the IN pattern to pull data. This process cannot verify scenarios where data exists in the destination but not in the source, and verification performance is degraded to a certain degree. If you select the default value Stop Migration, the full migration fails with an error if the destination table contains data. You must process the data in the destination table before you can restart the migration. Important If an error occurs and you click Resume, Data Transmission Service will ignore this configuration option and continue to migrate table data. Proceed with caution.
     Is deferred indexing supported?	You can set whether to allow index creation after full data migration is complete. The postponed index creation feature can shorten the time required for full migration. For notes on selecting postponed index creation, see the description below the table. Important On the Select Migration Type page, this parameter is displayed only if you select both Schema Migration and Full Migration. Only non-unique indexes support postponed creation. When creating an index, if the destination OceanBase database encounters the following errors, Data Transmission Service will ignore them and assume the index was created successfully, without attempting to create it again. OceanBase database in MySQL-compatible mode reports a Duplicate key name error. OceanBase database in Oracle-compatible mode reports a name is already used by an existing object error. If the destination is an OceanBase database and you select Allow, configure the following settings: Concurrency for a single index DDL statement: A higher degree of parallelism consumes more resources and results in a faster migration speed. Maximum number of concurrent index DDL statements: The maximum number of postponed index DDL statements that the system can call at the same time.

     If you allow postponed index creation, we recommend that you adjust the following business tenant parameters using a command-line tool according to the hardware of the OceanBase database and the current service traffic.

     // File memory buffer limit
     alter system set _temporary_file_io_area_size = '10' tenant = 'xxx'; 
     // V4.x Disable throttling
     alter system set sys_bkgd_net_percentage = 100;  

   • Incremental Synchronization

     The following parameters appear only when you select One-way Synchronization > Incremental Synchronization on the Select Migration Type page.

     

     Parameter	Description
     Write Concurrency	This parameter specifies the number of concurrent threads for writing data to the destination during incremental synchronization. The maximum value is 512. A high degree of concurrency may put excessive pressure on the destination and affect your services.
     Incremental Synchronization Rate Limit	You can decide whether to enable the incremental synchronization rate limit as needed. If you enable it, set RPS (the maximum number of data rows that can be synchronized to the destination per second during incremental synchronization) and BPS (the maximum volume of data that can be synchronized to the destination per second during incremental synchronization). Note The RPS and BPS settings here only serve as a rate-limiting capability. The actual performance of incremental synchronization is affected by factors such as the source, destination, and instance specifications.
     Incremental Synchronization Start Offset	If you select Full Migration, this parameter is not displayed. If you select Incremental Synchronization but not Full Migration, specify a point in time from which to start migrating data. The default value is the current system time. For more information, see Configure an incremental synchronization offset.

   • Reverse Incremental

     On the Select Migration Type page, the parameters for this area are displayed only if you select One-way Synchronization > Reverse Incremental. The configuration parameters for Reverse Incremental default to Reuse Incremental Synchronization Configuration.

     

     You can also choose not to reuse the incremental synchronization configuration and configure it manually.

     Parameter	Description
     Write Concurrency	This parameter specifies the number of concurrent threads for writing data to the source during reverse incremental synchronization. The maximum value is 512. A high degree of concurrency may put excessive pressure on the source and affect your services.
     Reverse Incremental Rate Limit	You can decide whether to enable the reverse incremental rate limit as needed. If you enable it, set RPS (the maximum number of data rows that can be synchronized to the source per second during reverse incremental synchronization) and BPS (the maximum volume of data that can be synchronized to the source per second during reverse incremental synchronization). Note The RPS and BPS settings here only serve as a rate-limiting capability. The actual performance of reverse incremental synchronization is affected by factors such as the source, destination, and instance specifications.
     Incremental Synchronization Start Offset	If you select Full Migration, this parameter is not displayed. If you do not select Full Migration but select Incremental Synchronization, the setting defaults to the forward switch (if any) and cannot be modified.

   • Advanced Options

     The parameters in this area are displayed only if the destination OceanBase database is in MySQL compatibility mode V4.3.0 or later and you select Schema Migration or Incremental Synchronization > DDL Synchronization on the Select Migration Type page.

     

     The available storage classes for destination table objects are Default, Row Store, Column Store, and Mixed Row and Column Store. This configuration determines the storage class for destination table objects created during schema migration or incremental synchronization. For more information, see default_table_store_format.

     Note

     The Default option automatically adjusts other settings based on the destination's parameters to ensure that table objects from a schema migration and new tables created by incremental DDL are written to the structure defined by the configured storage class.

8. Click Precheck to precheck the data migration task.

   In the Precheck step, Data Transmission Service checks whether items meet the requirements, such as the read and write permissions of the database user and the network connectivity of the database. You can start the data migration task only after all check items pass. If the precheck fails:

   • You can identify and troubleshoot the problem, and then run the precheck again until it succeeds.

   • You can also click Skip in the Actions column for a failed precheck item. A dialog box appears that describes the impact of skipping this operation. To proceed, click OK.

9. After the precheck passes, click Start Task.

   If you do not want to start the task immediately, click Save. You can then start the task manually from the Data Migration Task List page or using batch operations. For more information about batch operations, see Batch Operations on Data Migration Tasks.

   Data Transmission Service lets you modify migration objects and their row filtering conditions while a data migration task is running. For more information, see View and modify migration objects and their filter conditions. After the data migration task starts, it performs the migration steps sequentially based on the selected migration types. For more information, see View migration details.

References

• Create a data source

• Data migration

• Configure and modify matching rules

• View the details of a data migration task