Data synchronization between OceanBase Database tenants

Prerequisites

• You must grant Data Transmission Service the required access permissions to cloud resources. For more information, see Grant permissions for data migration to Data Transmission Service.

• You must create a database user for the data sync task in the OceanBase database and grant the required permissions to the user. For more information, see Configure data source permissions.

Limits

Data Transmission Service supports only objects with database, table, and column names that use ASCII characters. These names cannot contain special characters, such as line breaks, spaces, or any of the following: `.|"'`()=;/&\`.

Notes

• Clock skews between nodes, or between your computer and the server, can cause inaccurate latency reports for incremental synchronization.

  For example, if a clock is ahead of the standard time, the latency may be a negative value. If a clock is behind the standard time, the reported latency may be longer than the actual latency.

• If you select only Incremental Synchronization when you create a data sync task, the source database must retain its local incremental logs for more than 48 hours.

  If you select Full Synchronization and Incremental Synchronization when you create a data sync task, the source database must retain its local incremental logs for at least 7 days. Otherwise, Data Transmission Service may fail to obtain the incremental logs. This can cause the data sync task to fail and may result in data inconsistency between the source and the destination.

• If the source or destination contains table objects whose names differ only in case, the data synchronization results may be unexpected. This can happen if the source or destination is case-insensitive.

Supported source and destination instance types

Procedure

1. Log on to the OceanBase Management Console and purchase a data sync task.

   For more information, see Purchase a data sync task.

2. In the Data Transmission Service console, navigate to the Data Synchronization page, find the new sync task, and click Configure.

   

   If you want to reuse the configurations of an existing task, click Reuse Configuration. For more information, see Reuse and clear the configurations of a data sync task.

3. On the Select Source and Destination page, set the parameters.

   Parameter	Description
   Sync Task Name	Use a combination of Chinese characters, digits, and letters. The name cannot contain spaces and must be no more than 64 characters long.
   Source	If you have an existing OceanBase data source, select it from the drop-down list. If not, click Create Data Source in the drop-down list to create one in the dialog box that appears. For more information about the parameters, see Create an OceanBase data source.
   Destination	If you have an existing OceanBase data source, select it from the drop-down list. If not, click Create Data Source in the drop-down list to create one in the dialog box that appears.
   Tag (Optional)	Click the text box and select a tag from the drop-down list. You can also click Manage Tags to create, modify, or delete tags. For more information, see Manage data sync tasks using tags.

4. Click Next. On the Select Synchronization Types page, select the synchronization types.

   

   The synchronization types include Structure Synchronization, Full Synchronization, and Incremental Synchronization. Incremental Synchronization includes DML Synchronization (including INSERT, DELETE, and UPDATE) and DDL Synchronization. You can select the types as needed. For more information, see Customize DDL/DML and Scope of DDL synchronization.

5. Click Next. On the Select Sync Objects page, select the objects to sync.

   You can select sync objects using Specify Objects or Matching Rules. This topic describes how to select objects using Specify Objects. For more information about how to configure matching rules, see Configure and modify matching rules.

   Important

   • The names of the tables and columns to be synchronized cannot contain Chinese characters.

   • If a database or table name contains "$$", the data sync task may fail to be created.

   • If you selected DDL Synchronization in the Select Sync Type step, use matching rules to select the objects to synchronize. This ensures that all new objects that match the rules are included in the synchronization. If you specify objects directly instead of using rules, new or renamed objects will not be synchronized.

   

   1. In the Select Sync Objects section, select Specify Objects.

   2. In the Select Sync Scope section, select the objects to sync from the Source Objects list. You can select tables from one or more databases as sync objects.

   3. Click > to add them to the Destination Objects list.

   Data Transmission Service lets you import objects from a text file. You can also rename destination objects, set row filters, and remove a single object or all objects.

   Note

   When you use Matching Rules to select objects to sync, the matching rule syntax handles object renaming. You can only set filter conditions, select shard columns, and select columns to sync. For more information, see Configure and modify matching rules.

   Operation	Description
   Import Objects	In the list on the right, click Import Objects in the upper-right corner. In the dialog box, click OK. Important Importing will overwrite previous selections. Proceed with caution. In the Import Sync Objects dialog box, import the objects to synchronize. You can import a CSV file to rename database tables, set row filter conditions, and perform other operations. For more information, see Download and import sync object configurations. Click Check Validity. After the check is passed, click OK.
   Rename	Data Transmission Service lets you rename sync objects. For more information, see Rename database and table objects.
   Settings	Data Transmission Service supports row filtering with a WHERE clause. For more information, see Filter data with SQL conditions. You can also view the column information of sync objects in the View Columns section.
   Remove/Remove All	Data Transmission Service lets you remove one or more objects that have been temporarily selected for the destination during data mapping. Remove a single sync object In the list on the right, hover your mouse over the target object and click the Remove icon that appears. This removes the sync object. Remove all sync objects In the list on the right, click Remove All in the upper-right corner. In the dialog box, click OK to remove all sync objects.

6. Click Next. On the Sync Options page, configure the parameters.

   • Full synchronization

     The following parameters are displayed only if you select Full Synchronization on the Choose Sync Type page.

     

     Parameter	Description
     Read Concurrency	This parameter specifies the number of concurrent threads for reading data from the source during full synchronization. The maximum value is 512. A high concurrency can increase the load on the source database and affect your business.
     Write Concurrency	This parameter specifies the number of concurrent threads for writing data to the destination during full synchronization. The maximum value is 512. A high concurrency can increase the load on the destination database and affect your business.
     Full Sync Rate Limit	You can enable the full synchronization rate limit as needed. If you enable it, set the RPS and BPS. RPS is the maximum number of data rows that can be synchronized to the destination per second during full synchronization. BPS is the maximum volume of data that can be synchronized to the destination per second during full synchronization. Note The RPS and BPS settings only serve as a rate-limiting capability. The actual performance of full synchronization is affected by factors such as the source, the destination, and the instance specifications.
     Policy for Existing Records in Destination Table	Select Ignore: If data exists in the destination table and a conflict occurs between the existing data and the data being written, Data Transmission Service logs the conflict and keeps the existing data. Important If you select Ignore, full validation pulls data in IN mode. This mode cannot validate scenarios where extra data exists in the destination, and validation performance is degraded to some extent. Select the default value Stop Migration: If data exists in the destination table, an error is reported for the full migration. You must handle the data in the destination before you can continue the migration. Important If an error occurs and you click Resume, Data Transmission Service ignores this setting and continues to migrate table data. Proceed with caution.

   • Incremental synchronization

     The following parameters are displayed only if you select Incremental Synchronization on the Choose Synchronization Types page.

     

     Parameter	Description
     Write Concurrency	This parameter specifies the number of concurrent threads for writing data to the destination during incremental synchronization. The maximum value is 512. A high concurrency may put excessive pressure on the destination and affect your business.
     Incremental Synchronization Rate Limit	You can decide whether to enable the incremental synchronization rate limit as needed. If you enable it, set RPS (the maximum number of data rows that can be synchronized to the destination per second during incremental synchronization) and BPS (the maximum data volume that can be synchronized to the destination per second during incremental synchronization). Note The RPS and BPS values set here are only for rate limiting. The actual performance of incremental synchronization is affected by factors such as the source, destination, and instance specifications.
     Incremental Synchronization Start Offset	If you select Full Synchronization when you select the synchronization type, you cannot change this parameter. If you do not select Full Synchronization but select Incremental Synchronization when you select the synchronization type, specify a point in time after which you want to synchronize data. The default value is the current system time. For more information, see Set the start offset for incremental synchronization.

   • Advanced options

     

     Parameter	Description
     Storage Class for Target Table Objects	This parameter is displayed only if the destination OceanBase database is V4.3.0 or later and you select Structure Synchronization or Incremental Synchronization > DDL Synchronization on the Select Sync Type page. The storage classes for target table objects are Default, Row store, Column store, and Hybrid row-column store. This setting defines the storage class for target table objects during structure or incremental synchronization. For more information, see default_table_store_format. Note The Default option automatically uses the parameter settings from the destination. The structure of table objects for structure synchronization or new table objects for incremental DDL is created based on the specified storage class.
     Add Hidden Columns to Tables Without Non-null Unique Keys	For data synchronization between OceanBase Database tenants, this parameter is displayed only if you select Structure Synchronization or Incremental Synchronization > DDL Synchronization on the Select Sync Type page. It configures whether to add hidden columns to tables without non-null unique keys. For more information, see the hidden column mechanism of Data Transmission Service.

7. Click Precheck.

   In the Precheck step, Data Transmission Service checks for several items, such as the required read and write permissions for the database user and normal network connectivity. The data sync task can start only after all precheck items pass. If the precheck fails:

   • You can identify and fix the problem, and then rerun the precheck until it is successful.

   • You can also click Skip in the Actions column for a failed precheck item. A dialog box appears to inform you of the impact of skipping this item. To confirm, click OK.

8. After the precheck passes, click Start Task.

   If you do not need to start the task right away, click Save. You can then start the task manually on the Sync Task List page or include it in a batch operation. For more information about batch operations, see Perform batch operations on data sync tasks.

   DTS lets you modify the synchronization objects while a data sync task is running. For more information, see View and modify synchronization objects and their filter conditions. After the data sync task starts, it runs based on the selected synchronization types. For more information, see View synchronization details.

If a data sync task fails, typically due to network connection issues or a slow process startup, you can click Resume on the task list or the product page.

References

• Create an OceanBase data source

• Perform batch operations on data sync tasks

• View the details of a data sync task