Migrate or synchronize data within a time range

Prerequisites

• You have verified the network connectivity between the source cluster, the destination cluster, and LTS.

• You have added a Lindorm data source. For more information, see Add a Lindorm wide table data source.

Usage notes

• Before you start a migration, check the HDFS capacity of the destination instance. Ensure that sufficient space is available to prevent task failures.

• Filtering data by time range requires data parsing during read operations. This process consumes CPU resources and may increase the data migration time.

• Before you create a real-time synchronization task, check the log retention period (hbase.master.logcleaner.ttl) of the source cluster to prevent task failures due to a short log retention period. For more information about how to check and modify the retention period, see Modify the log retention period.

  Note

  A real-time synchronization task is created by either creating a Real-time Data Replication task or selecting Real-time Data Replication when you create a One-click Migration task.

Historical data migration

1. Log on to the LTS console.

2. In the left-side navigation pane, choose Lindorm/HBase Migration > Historical Data Migration.

3. Click Create Task.

4. Specify the Source Cluster and Destination Cluster, and select schema migration.

5. In the Tables to Synchronize/Migrate field, add a time parameter in the format <table_name>{"startTs": "<start_timestamp>", "endTs": "<end_timestamp>"}. For example, tableA {"startTs": "20200101000000", "endTs": "20200102000000"} migrates the data in tableA from 00:00:00 on January 1, 2020, to 00:00:00 on January 2, 2020.

   Parameter	Description
   startTs	The start time specifies that data at and after this point in time is migrated. The format is YYYYMMDDHHMMSS, which represents YearMonthDayHourMinuteSecond.
   endTs	The end time specifies that data created up to and including this point in time is migrated. The format is YYYYMMDDHHMMSS, which represents YearMonthDayHourMinuteSecond.

   Important

   startTs and endTs represent a specific point in time. The format is not a timestamp. For example, 20200101132250 represents 13:22:50 on January 1, 2020.

   You can specify startTs and endTs based on your requirements:

   • Specify only a start time: Set only the startTs parameter in the format <table_name>{"startTs": "<start_time>"}. This migrates all historical data after the specified point in time.

   • Specify only the end time point: You only need to specify the endTs parameter. The format is <table_name>{"endTs": "<end_time_point>"}. This migrates all historical data before the specified time point.

   Navigate to Lindorm/HBase Migration > Historical Data Migration to open the Lindorm/HBase Full Data Synchronization page. On this page, select schema migration if needed, enter the table name and time parameters in the Tables to Synchronize/Migrate field, and then click Create.

6. (Optional) If the source cluster is a self-managed HBase cluster, in Advanced settings, enter {"exporterEnable":false}. If the source cluster is not a self-managed cluster, you can skip this step.

7. Click Create to create the migration task.

Real-time data replication

1. In the left-side navigation pane of the LTS console, choose Lindorm/HBase Migration > Real-time Data Replication.

2. Click Create Task.

3. Specify the Source Cluster, Destination Cluster, and Tables to Synchronize/Migrate, and select schema migration.

4. In Advanced settings, enter the time parameters in the format {"startTS": <startTimestamp>,"endTS":<endTimestamp>} to synchronize real-time data within a time range.

   Parameter	Description
   startTS	The start time. Data at or after this time is synchronized. startTS must be a 13-digit timestamp in milliseconds (ms). Important Real-time data replication uses the Write-Ahead Log (WAL) to synchronize data. Therefore, the specified startTS must be greater than or equal to the current time.
   endTS	The end time. Data at or before this time is synchronized. endTS must be a 13-digit timestamp in milliseconds (ms).

   You can specify startTS and endTS based on your requirements:

   • To specify only the synchronization start time, set startTS in the format {"startTS": "<start_timestamp>"}. This synchronizes all real-time data after the specified timestamp.

   • Specify only the synchronization end time: Set the endTS parameter in the format {"endTS": "<end_timestamp>"}. This starts data synchronization from the current time and ends it at the specified timestamp.

   Important

   If the source cluster is a self-managed HBase cluster, you also need to add "exporterEnable":false in the Advanced Settings.

   On the Lindorm/HBase Real-time Data Synchronization page, specify the Task Name, Source Cluster, and Destination Cluster. In the Operations section, select schema migration. In the Tables to Synchronize/Migrate field, enter the names of the tables that you want to synchronize. Before you submit the task, confirm the log retention period (hbase.master.logcleaner.ttl) of the source cluster to prevent task failure due to a short retention period. After you complete the configuration, click Create.

5. Click Create to create the synchronization task.

One-click migration

You can use One-click Migration to configure both historical migration and real-time synchronization tasks.

1. In the left-side navigation pane of the LTS console, choose Lindorm/HBase Migration > One-click Migration.

2. Click Create Task.

3. Configure the following settings as needed: Source Cluster, Destination Cluster, schema migration, Real-time Data Replication, and Historical Data Migration.

4. In the Tables to Synchronize/Migrate field, specify the time range for historical data migration. For more information, see Configure the migration time range.

5. In the advanced settings, specify the time range for real-time synchronization. For more information, see Configure the synchronization time range.

   Important

   If the source cluster is a self-managed HBase cluster, you also need to add "exporterEnable":false in Advanced settings.

6. Click Create to create the task.