Phoenix full and incremental migration-ApsaraDB for HBase(Hbase)-阿里云帮助中心

Use cases

Cross-region migration: For example, migrating from a cluster in the China (Qingdao) region to one in the China (Beijing) region.
Cluster scale-up: For example, migrating from a 4-core, 8 GB cluster to an 8-core, 16 GB cluster.
Service isolation: Migrating a subset of business services to a new cluster.

Features

Migrations are supported only between identical Phoenix versions. Cross-version migration, such as from Phoenix 4.x to Phoenix 5.x, is not supported.
Supports table structure migration, real-time data synchronization, and full data migration.
Supports migration at the database, namespace, and table levels.
Allows you to rename tables during migration.
Allows you to migrate data by specifying a time range, rowkey range, or specific columns.
Provides an OpenAPI to create migration tasks programmatically.

Benefits

Live data migration without service interruption. You can migrate historical data and synchronize real-time incremental data in a single process.
Minimized impact on the source cluster's online services. The migration process reads data directly from HDFS without interacting with the source HBase service.
High efficiency. Data is copied at the file level, which typically uses over 50% less network bandwidth than API-level data migration.
High performance and scalability. A single node can achieve migration speeds of up to 150 MB/s. You can horizontally scale out nodes to support stable terabyte- and petabyte-scale data migrations.
Reliability and monitoring. Includes a robust error retry mechanism, real-time monitoring of task speed and progress, and alerts for task failures.
Automatic schema synchronization ensures partition consistency between the source and destination clusters.

Limitations

Kerberos-enabled clusters are not supported.
Single-node ApsaraDB for HBase instances are not supported.
Due to network restrictions, ApsaraDB for HBase instances in a Classic Network are not supported.
Incremental data synchronization relies on asynchronously replicating the HBase WAL. Data loaded using BulkLoad or any method that bypasses the WAL will not be synchronized.

Notes

Before starting the migration, verify that the destination cluster has sufficient HDFS capacity to avoid running out of storage during the migration.
Before starting incremental data synchronization, increase the log retention period on the source cluster. This provides more time to troubleshoot potential synchronization errors. Set the hbase.master.logcleaner.ttl property in the hbase-site.xml file to 12 hours or more, and then restart the HMaster process.
Do not create tables on the destination cluster. The LTS synchronization service automatically creates tables that match the schema and partition configuration of the source cluster. If you manually create tables, their partition schemes might be inconsistent with the source tables. This can lead to frequent and time-consuming split and compaction operations on the destination cluster after migration, especially for large tables.
If a source table has a coprocessor, ensure that the corresponding coprocessor JAR file is available on the destination cluster before the table is created.

Before you begin

Verify the network connectivity between the source cluster, the destination cluster, and Lindorm Tunnel Service (LTS).
Add a Phoenix data source.
Log on to the LTS console.

Create a task

In the left-side navigation pane, choose Lindorm/HBase Migration > Phoenix -> Phoenix to navigate to the Phoenix live migration page.
Click Create Task.

Table Mapping supports the following formats: * migrates all tables; schema.* migrates all tables within a specific schema; schema.tableName migrates a specific table within a schema; and tableName migrates a specific table that does not belong to a schema.

Task Name: Optional. The name can contain only letters and digits. If you leave this empty, the system uses the task ID as the name.
Follow the on-screen instructions to configure the Source Cluster and Destination Cluster.
Select the operations to perform:
- Table Structure Migration: Creates tables on the destination cluster with matching schemas and partitions. If a table with the same name already exists, the service skips it.
- Real-time Data Synchronization: Synchronizes incremental data changes from the source cluster.
- Full Data Migration: Performs a file-level physical migration of all historical data.

Table Mapping: Enter the names of the tables to migrate, separated by line breaks.

Advanced Configuration: Optional.

View task details

In the left-side navigation pane, choose Lindorm/HBase Migration > Phoenix -> Phoenix.
On the Phoenix Live Migration page, view your tasks in the Running Tasks and Completed Tasks tables. The tables include columns such as Task Name, Source Cluster, Destination Cluster, Task Progress, Status, Start Time, and Operations. Click the Task Name of a task to view its details.

Review the execution status of the task.

The task details page contains three sections: The DataX-like tasks section displays the task name, status, task progress, start time, end time, and available operations such as Retry and Details. The Real-time tasks section shows the channel name, source and destination clusters, channel status, synchronization latency, task queue, and synchronization checkpoint. The Batch migration tasks section displays the total number of subtasks and the counts of completed and failed subtasks. You can filter the list by status: All, Queuing, Running, Completed, or Failed. The columns in this section include the task name, table name, source and destination clusters, status, and task progress.

Cutover

Wait until the full data migration is complete and the incremental data synchronization latency is low (typically a few seconds or less).
Use the LTS data sampling and verification feature. For large tables, use a small sampling ratio to avoid affecting online services.
Verify your business applications on the destination cluster.
Perform the application cutover to the destination cluster.