During data migration, high migration speeds can overload the destination cluster and degrade system performance. High speeds can also increase network latency, which may lead to data loss and compromise data integrity. To prevent system overload and ensure data consistency, you can rate-limit migration tasks in the LTS console.
Background information
LTS splits migration and synchronization tasks into multiple threaded tasks for worker nodes to execute. Therefore, the total rate limit is calculated as follows: Rate limit per thread * Number of threads per worker * Total number of worker nodes.
The total number of worker nodes is the number of LTS nodes that you specified during purchase. You can view the total number of worker nodes in the LTS console. For more information, see View the total number of worker nodes.
Prerequisites
-
You have verified the network connectivity between the source cluster, destination cluster, and LTS.
-
You have added a Lindorm data source. For instructions, see Add a Lindorm wide table data source.
Precautions
Before you start a data migration, confirm that the destination instance has sufficient HDFS capacity. This prevents task failures due to insufficient storage during the migration.
View worker nodes
-
In the left-side navigation pane, click Cluster Information.
-
In the Basic Information section, view the total number of worker nodes. You can also view details about the worker nodes for the current instance in the Worker Node Information section.
The details include the Node Name, Start Time, Task Information (Total/Running), and Load (CPU and Memory percentages).
Global configuration
You can use the global configuration to rate-limit all historical data migration tasks, or you can configure advanced parameters when creating a migration task to rate-limit a single task.
-
In the upper-right corner of the page, click Configuration Management.
-
On the Configuration Management page, modify the current value of the following configuration items:
-
bds.full.reader.bandwidth: The maximum bandwidth for a single-threaded task, in bytes/s.
-
bds.full.task.pool.size: The number of threaded tasks that each worker node can process.
-
-
Click Modify in the lower-left corner to save the configuration. The default value for
bds.full.reader.bandwidthis10485760(10 MB/s), and the default value forbds.full.task.pool.sizeis5. After modifying the values, click Modify.
Rate-limit a single migration task
You can rate-limit a single migration task in the following two ways:
-
Maximum bandwidth per thread: You configure this by using advanced parameters when you create a migration task. This setting applies only to the specific task and overrides the global configuration.
-
Number of threads per worker: This is a global setting that you must modify in Configuration Management.
Limit bandwidth per thread
-
In the left-side navigation pane of the LTS console, choose .
-
Click Create Task.
-
Configure the required parameters: Source Cluster, Destination Cluster, Schema Migration, and Tables to Synchronize/Migrate.
-
In the Advanced Settings section, specify the bandWidth parameter. The bandWidth parameter is specified in bytes/s. For example, to set the maximum bandwidth to 10 MB/s, enter
{"bandWidth": 10485760}. -
Click Create to create the task and save the configuration.
Limit threads per worker
The number of threads per worker node (bds.full.task.pool.size) cannot be configured in the Advanced Settings for an individual task. You must modify this setting in Configuration Management. To do so, follow these steps:
-
In the upper-right corner of the LTS console, click Configuration Management.
-
On the Configuration Management page, modify the current value of bds.full.task.pool.size.
-
Click Modify to save the configuration.