Create DTS tasks using Alibaba Cloud CLI-阿里云帮助中心

This topic shows how to use Alibaba Cloud CLI to create and configure Data Transmission Service (DTS) tasks for data migration, data synchronization, data subscription, and data validation.

Overview

Creating a DTS task using Alibaba Cloud CLI involves three main steps:

Purchase a DTS instance: Call the CreateDtsInstance operation to purchase a DTS instance of a specified type and get its instance ID.
Configure the DTS task: Call the ConfigureDtsJob operation to configure the connection information for the source and destination databases, the objects to migrate or synchronize, and other settings.
Start the DTS task: Call the StartDtsJob operation to start the task, and then call the DescribeDtsJobDetail operation to query its status.

The examples in this topic demonstrate transferring data from an RDS for MySQL source database to an RDS for MySQL destination database. The examples show the CLI commands to create data migration, data synchronization, data subscription, and data validation tasks. You can replace the parameters for the source and destination databases.

Prerequisites

You have installed and configured Alibaba Cloud CLI. For more information, see Install or update Alibaba Cloud CLI and Configure and manage credentials.
The RAM user for your AccessKey must have the required permissions to manage DTS. For more information, see Grant permissions to a RAM user to manage DTS by using system policies.
You have created the source and destination database instances and completed the necessary preparations for DTS. For more information, see Preparations.

Notes

The SourceRegion and DestinationRegion parameters in the CreateDtsInstance operation must match the regions of your source and destination databases.
The Type parameter in the CreateDtsInstance operation specifies the task type. Valid values: MIGRATION (data migration), SYNC (data synchronization), and SUBSCRIBE (data subscription).
The DbList parameter in the ConfigureDtsJob operation is in JSON format and specifies the objects to migrate or synchronize. For format details, see Objects to migrate, synchronize, or subscribe to.
Data migration tasks support only the pay-as-you-go billing method. Data synchronization and data subscription tasks support both pay-as-you-go and subscription billing methods.
Pass database password parameters, such as SourceEndpointPassword and DestinationEndpointPassword, as environment variables to avoid exposing them in plaintext on the command line.

Create a data migration task

The following example shows how to use Alibaba Cloud CLI to create a data migration task from an RDS for MySQL database to another.

Step 1: Purchase a migration instance

Run the following command to call the CreateDtsInstance operation to purchase a data migration instance.

aliyun dts CreateDtsInstance \
  --Type MIGRATION \
  --SourceRegion cn-hangzhou \
  --DestinationRegion cn-shanghai \
  --SourceEndpointEngineName MySQL \
  --DestinationEndpointEngineName MySQL \
  --PayType PostPaid \
  --InstanceClass small

Sample output:

{
  "InstanceId": "dtsse7s11v26j****",
  "JobId": "se7s11v26j****",
  "RequestId": "E52C5BD7-AF7D-4F5D-A8F5-D3B5F0****",
  "Success": true
}

Record the InstanceId from the returned result. This value is the DTS instance ID and is used as the DtsInstanceId parameter in subsequent steps.

Step 2: Configure the migration task

Run the following command to call the ConfigureDtsJob operation to configure the source and destination databases.

aliyun dts ConfigureDtsJob \
  --DtsInstanceId dtsse7s11v26j**** \
  --DtsJobName "rds_mysql_migration_demo" \
  --JobType MIGRATION \
  --SourceEndpointInstanceType RDS \
  --SourceEndpointInstanceID rm-bp1imrtn6fq****  \
  --SourceEndpointEngineName MYSQL \
  --SourceEndpointRegion cn-hangzhou \
  --SourceEndpointDatabaseName source_db \
  --SourceEndpointUserName dts_user \
  --SourceEndpointPassword '${SOURCE_DB_PASSWORD}' \
  --DestinationEndpointInstanceType RDS \
  --DestinationEndpointInstanceID rm-bp1k5v0rm3d****  \
  --DestinationEndpointEngineName MYSQL \
  --DestinationEndpointRegion cn-shanghai \
  --DestinationEndpointDataBaseName target_db \
  --DestinationEndpointUserName dts_user \
  --DestinationEndpointPassword '${DEST_DB_PASSWORD}' \
  --StructureInitialization true \
  --DataInitialization true \
  --DataSynchronization true \
  --DbList '{"source_db":{"name":"source_db","all":true}}'

Sample output:

{
  "RequestId": "1D234B10-5C84-4F26-B06A-3E48D2****",
  "DtsJobId": "se7s11v26j****",
  "ErrCode": "",
  "Success": "true",
  "HttpStatusCode": 200
}

Step 3: Start the migration task

Run the following command to call the StartDtsJob operation to start the migration task.

aliyun dts StartDtsJob \
  --DtsJobId se7s11v26j****

Sample output:

{
  "RequestId": "A7B3E61D-2C2F-4B3B-A5D8-F7E49D****",
  "ErrCode": "",
  "Success": "true",
  "HttpStatusCode": 200
}

Step 4: Query the task status

Run the following command to call the DescribeDtsJobDetail operation to query the status of the migration task.

aliyun dts DescribeDtsJobDetail \
  --DtsJobId se7s11v26j****

Confirm that the Status in the returned result is Migrating, which indicates that the migration task is running.

Create a data synchronization task

The following example shows how to use Alibaba Cloud CLI to create a data synchronization task from an RDS for MySQL database to another.

Step 1: Purchase a synchronization instance

Run the following command to call the CreateDtsInstance operation to purchase a data synchronization instance.

aliyun dts CreateDtsInstance \
  --Type SYNC \
  --SourceRegion cn-hangzhou \
  --DestinationRegion cn-hangzhou \
  --SourceEndpointEngineName MySQL \
  --DestinationEndpointEngineName MySQL \
  --PayType PostPaid \
  --InstanceClass small \
  --SyncArchitecture oneway

Sample output:

{
  "InstanceId": "dtsbi6e22ay24****",
  "JobId": "bi6e22ay24****",
  "RequestId": "6C2FE2A1-B9D3-4E5C-8A1B-C3F28E****",
  "Success": true
}

Step 2: Configure the synchronization task

Run the following command to call the ConfigureDtsJob operation to configure the source and destination databases for the synchronization task.

aliyun dts ConfigureDtsJob \
  --DtsInstanceId dtsbi6e22ay24**** \
  --DtsJobName "rds_mysql_sync_demo" \
  --JobType SYNC \
  --SourceEndpointInstanceType RDS \
  --SourceEndpointInstanceID rm-bp1imrtn6fq****  \
  --SourceEndpointEngineName MYSQL \
  --SourceEndpointRegion cn-hangzhou \
  --SourceEndpointDatabaseName source_db \
  --SourceEndpointUserName dts_user \
  --SourceEndpointPassword '${SOURCE_DB_PASSWORD}' \
  --DestinationEndpointInstanceType RDS \
  --DestinationEndpointInstanceID rm-bp1k5v0rm3d**** \
  --DestinationEndpointEngineName MYSQL \
  --DestinationEndpointRegion cn-hangzhou \
  --DestinationEndpointDataBaseName target_db \
  --DestinationEndpointUserName dts_user \
  --DestinationEndpointPassword '${DEST_DB_PASSWORD}' \
  --StructureInitialization true \
  --DataInitialization true \
  --DataSynchronization true \
  --DbList '{"source_db":{"name":"source_db","all":true}}'

Sample output:

{
  "RequestId": "8E2D3A9C-4B5F-4C6A-9D7E-A1B2C3****",
  "DtsJobId": "bi6e22ay24****",
  "ErrCode": "",
  "Success": "true",
  "HttpStatusCode": 200
}

Step 3: Start the synchronization task

Run the following command to call the StartDtsJob operation to start the synchronization task.

aliyun dts StartDtsJob \
  --DtsJobId bi6e22ay24****

Step 4: Query the task status

Run the following command to call the DescribeDtsJobDetail operation to query the status of the synchronization task.

aliyun dts DescribeDtsJobDetail \
  --DtsJobId bi6e22ay24****

Confirm that the Status in the response is Synchronizing, which indicates that the synchronization task is in progress.

aliyun dts CreateDtsInstance \
  --Type SUBSCRIBE \
  --SourceRegion cn-hangzhou \
  --DestinationRegion cn-hangzhou \
  --SourceEndpointEngineName MySQL \
  --PayType PostPaid \
  --InstanceClass small

{
  "InstanceId": "dtsk3m1zb527a****",
  "JobId": "k3m1zb527a****",
  "RequestId": "3F5A8B2C-6D7E-4A9B-8C1D-E2F3A4****",
  "Success": true
}

aliyun dts ConfigureSubscription \
  --RegionId cn-hangzhou \
  --DtsInstanceId dtsk3m1zb527a**** \
  --DtsJobName "rds_mysql_subscribe_demo" \
  --SourceEndpointInstanceType RDS \
  --SourceEndpointInstanceID rm-bp1imrtn6fq**** \
  --SourceEndpointEngineName MYSQL \
  --SourceEndpointRegion cn-hangzhou \
  --SourceEndpointDatabaseName source_db \
  --SourceEndpointUserName dts_user \
  --SourceEndpointPassword '${SOURCE_DB_PASSWORD}' \
  --DbList '{"source_db":{"name":"source_db","all":true}}' \
  --SubscriptionDataTypeDDL true \
  --SubscriptionDataTypeDML true \
  --SubscriptionInstanceNetworkType vpc \
  --SubscriptionInstanceVPCId vpc-bp1opxu1zkhn00g****  \
  --SubscriptionInstanceVSwitchId vsw-bp1s5fnvk4gn2tw****

{
  "RequestId": "B4C5D6E7-8F9A-4B1C-2D3E-F4A5B6****",
  "DtsJobId": "k3m1zb527a****",
  "ErrCode": "",
  "Success": "true",
  "HttpStatusCode": 200
}

aliyun dts StartDtsJob \
  --DtsJobId k3m1zb527a****

aliyun dts DescribeDtsJobDetail \
  --DtsJobId k3m1zb527a****

Verify that the Status in the response is Normal, which indicates that the subscription task is running.

Create a data validation task

A data validation task verifies data consistency between the source and destination databases after data migration or synchronization. The following example shows how to create a validation task for an existing data migration task.

Step 1: Purchase a validation instance

Run the following command to call the CreateDtsInstance operation to purchase a data validation instance.

aliyun dts CreateDtsInstance \
  --Type CHECK \
  --SourceRegion cn-hangzhou \
  --DestinationRegion cn-shanghai \
  --SourceEndpointEngineName MySQL \
  --DestinationEndpointEngineName MySQL \
  --PayType PostPaid \
  --InstanceClass small

Sample output:

{
  "InstanceId": "dtsp2w8rx6315****",
  "JobId": "p2w8rx6315****",
  "RequestId": "D5E6F7A8-9B0C-4D1E-2F3A-B4C5D6****",
  "Success": true
}

Step 2: Configure the validation task

Run the following command to call the ConfigureDtsJob operation to configure the source and destination databases for the validation task.

aliyun dts ConfigureDtsJob \
  --DtsInstanceId dtsp2w8rx6315**** \
  --DtsJobName "rds_mysql_check_demo" \
  --JobType CHECK \
  --SourceEndpointInstanceType RDS \
  --SourceEndpointInstanceID rm-bp1imrtn6fq**** \
  --SourceEndpointEngineName MYSQL \
  --SourceEndpointRegion cn-hangzhou \
  --SourceEndpointDatabaseName source_db \
  --SourceEndpointUserName dts_user \
  --SourceEndpointPassword '${SOURCE_DB_PASSWORD}' \
  --DestinationEndpointInstanceType RDS \
  --DestinationEndpointInstanceID rm-bp1k5v0rm3d**** \
  --DestinationEndpointEngineName MYSQL \
  --DestinationEndpointRegion cn-shanghai \
  --DestinationEndpointDataBaseName target_db \
  --DestinationEndpointUserName dts_user \
  --DestinationEndpointPassword '${DEST_DB_PASSWORD}' \
  --StructureInitialization false \
  --DataInitialization true \
  --DataSynchronization false \
  --DbList '{"source_db":{"name":"source_db","all":true}}'

Sample output:

{
  "RequestId": "E6F7A8B9-0C1D-4E2F-3A4B-C5D6E7****",
  "DtsJobId": "p2w8rx6315****",
  "ErrCode": "",
  "Success": "true",
  "HttpStatusCode": 200
}

Step 3: Start the validation task

Run the following command to call the StartDtsJob operation to start the validation task.

aliyun dts StartDtsJob \
  --DtsJobId p2w8rx6315****

Step 4: Query the validation results

Run the following command to call the DescribeDtsJobDetail operation to query the results of the validation task.

aliyun dts DescribeDtsJobDetail \
  --DtsJobId p2w8rx6315****

Confirm that the Status in the returned result is Finished. This indicates that the verification task is complete. You can view the verification details in the returned result to determine whether the data is consistent.

API	Description
CreateDtsInstance	Purchases a DTS instance.
ConfigureDtsJob	Configures a data migration, data synchronization, or data validation task.
ConfigureSubscription	Configures a data subscription task.
StartDtsJob	Starts a DTS task.
DescribeDtsJobDetail	Queries the details of a DTS task.
DescribeDtsJobs	Queries a list of DTS tasks and the details of each task.

Overview

Prerequisites

Notes

Create a data migration task

Step 1: Purchase a migration instance

Step 2: Configure the migration task

Step 3: Start the migration task

Step 4: Query the task status

Create a data synchronization task

Step 1: Purchase a synchronization instance

Step 2: Configure the synchronization task

Step 3: Start the synchronization task

Step 4: Query the task status

Create a data subscription task

Step 1: Purchase a subscription instance

Step 2: Configure the subscription task

Step 3: Start the subscription task

Step 4: Query the task status

Create a data validation task

Step 1: Purchase a validation instance

Step 2: Configure the validation task

Step 3: Start the validation task

Step 4: Query the validation results

Related APIs