Sample scenarios for Data Online Migration SDK for Python

Updated at:
Copy as MD

Data Online Migration SDK for Python supports two common scenarios: running a complete migration task from scratch and retrying failed files from a previous migration.

Prerequisites

  • An Alibaba Cloud account with an AccessKey pair. To obtain an AccessKey pair, see Create an AccessKey pair.

  • Python 3.6 or later.

  • The Data Online Migration SDK for Python is installed:

      pip install alibabacloud_hcs_mgw20240626
      pip install alibabacloud_tea_openapi
  • Environment variables ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET are configured.

Scenario 1: Run a migration task from scratch

The following example shows the complete workflow for creating and running a migration task:

  1. (Optional) Create a tunnel and an agent. Skip this step if you do not need an agent.

  2. Create source and destination data addresses.

  3. Create a migration task.

  4. Start the task and view the results.

Note

If you do not use an agent, skip the tunnel and agent creation in the following code. You must deploy an agent before using it. For more information, see Manage agents. A migration task is referred to as a job in the API. For example, the create_job method creates a migration task.

import os
import sys
import time

from alibabacloud_hcs_mgw20240626.client import Client
from alibabacloud_hcs_mgw20240626.models import *
from alibabacloud_tea_openapi.models import Config

if __name__ == "__main__":
    try:
        # The ID of your Alibaba Cloud account.
        userid = "11470***876***55"
        # In this example, the endpoint in the China (Beijing) region is used.
        endpoint = "cn-beijing.mgw.aliyuncs.com"

        """
        Step 1: Initialize a client.
        """
        # Supports HTTPS and HTTP. If not specified, HTTPS is used by default.
        config = Config(
            endpoint=endpoint,
            access_key_id=os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_ID"),
            access_key_secret=os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_SECRET"),
            protocol="http"
        )
        client = Client(config)

        """
        Step 2: (Optional) Create a tunnel and one or more agents.
        The agents can be associated with the same tunnel.
        """
        # The default values of max_bandwidth and max_qps are 0,
        # which means bandwidth and queries per second (QPS) are not limited.
        # Unit of max_bandwidth: bit/s.
        max_bandwidth = 1073741824
        max_qps = 1000

        resp = client.create_tunnel(userid, CreateTunnelRequest(
            CreateTunnelInfo(tunnel_qos=TunnelQos(max_bandwidth=max_bandwidth, max_qps=max_qps))))
        print("Tunnel ID:" + resp.headers["x-oss-import-tunnel-id"])
        tunnel_id = resp.headers["x-oss-import-tunnel-id"]

        # The name of the agent.
        agent_name = "exampleagent"
        # The network type. Valid values:
        # - public: the Internet.
        # - vpc: an Express Connect circuit or a VPN gateway.
        agent_endpoint = "public"
        # The deployment method. Set the value to default.
        deploy_method = "default"
        client.create_agent(userid, CreateAgentRequest(CreateAgentInfo(
            agent_endpoint=agent_endpoint,
            deploy_method=deploy_method,
            name=agent_name,
            tunnel_id=tunnel_id
        )))

        """
        Step 3: Create a source data address and a destination data address,
        and verify their availability.
        The following example creates data addresses of the Object Storage Service (OSS) type.
        For more information about other types, see the "Create a data address" section
        of the "Manage data addresses" topic.
        """
        # The name of the source data address.
        src_address = "examplesrcaddress"
        src_address_type = "oss"
        # The OSS region ID, such as oss-cn-beijing.
        src_region_id = "oss-cn-beijing"
        src_bucket = "examplebucket"
        src_prefix = "***/"
        src_role_name = "rolename_******"
        # The agents to be associated with the tunnel.
        # Separate multiple agent names with commas (,).
        # Skip this part if you do not use agents.
        agent_list = agent_name
        src_detail = AddressDetail(address_type=src_address_type,
                                   region_id=src_region_id,
                                   bucket=src_bucket,
                                   prefix=src_prefix,
                                   role=src_role_name)
        client.create_address(userid, CreateAddressRequest(CreateAddressInfo(name=src_address, address_detail=src_detail)))
        response = client.verify_address(userid, src_address)
        if response.body.verify_address_response.status != "available":
            print("verify address status failed, status: ", response.body.verify_address_response.status)
            sys.exit()

        # The name of the destination data address.
        dest_address = "exampledestaddress"
        dest_address_type = "oss"
        # The OSS region ID, such as oss-cn-beijing.
        dest_region_id = "oss-cn-beijing"
        dest_bucket = "examplebucket"
        dest_prefix = "***/"
        dest_role_name = "rolename_******"
        dest_detail = AddressDetail(address_type=dest_address_type,
                                   region_id=dest_region_id,
                                   bucket=dest_bucket,
                                   prefix=dest_prefix,
                                   role=dest_role_name)
        client.create_address(userid, CreateAddressRequest(CreateAddressInfo(name=dest_address, address_detail=dest_detail)))
        response = client.verify_address(userid, dest_address)
        if response.body.verify_address_response.status != "available":
            print("verify address status failed, status: %s" % response.body.verify_address_response.status)
            sys.exit()

        """
        Step 4: Create and start a data migration task.
        """
        # The name of the task.
        job_name = "examplejob"
        # Specify the following parameters based on your business requirements.
        overwrite_mode = "always"
        transfer_mode = "lastmodified"

        # Specify max_band_width and max_import_task_qps based on your business requirements.
        max_import_task_qps = 1000
        max_band_width = 2147483648
        # If you set these parameters to 0 or leave them empty, the default values are used.
        # Unit of max_band_width: bit/s.
        import_qos = ImportQos(max_import_task_qps=max_import_task_qps, max_band_width=max_band_width)

        # Configure file type, file key, and time filters.
        # For more information about parameters, see the API reference.

        # The file type filter. This filter is suitable for local file systems.
        exclude_symlink = False
        exclude_dir = False
        file_type_filters = FileTypeFilters(exclude_dir=exclude_dir, exclude_symlink=exclude_symlink)

        # The file key filter. Configure based on your business requirements.
        include_regex = [".*.jpg", ".*.gif"]
        exclude_regex = ["txtFile", ".*.js"]
        exclude_regex_item = KeyFilterItem(regex=exclude_regex)
        include_regex_item = KeyFilterItem(regex=include_regex)
        key_filters = KeyFilters(excludes=exclude_regex_item, includes=include_regex_item)

        # The time filter. Times must be in UTC.
        include_start_time = "2006-01-01T00:00:00Z"
        include_end_time = "2007-01-01T00:00:00Z"
        exclude_start_time = "2009-01-01T00:00:00Z"
        exclude_end_time = "2010-01-01T00:00:00Z"
        include_time_filters = [TimeFilter(start_time=include_start_time, end_time=include_end_time)]
        exclude_time_filters = [TimeFilter(start_time=exclude_start_time, end_time=exclude_end_time)]
        include_lastmodified_filter_item = LastModifyFilterItem(time_filter=include_time_filters)
        exclude_lastmodified_filter_item = LastModifyFilterItem(time_filter=exclude_time_filters)
        last_modified_filters = LastModifiedFilters(excludes=exclude_lastmodified_filter_item,
                                                    includes=include_lastmodified_filter_item)

        filter_rule = FilterRule(file_type_filters=file_type_filters,
                                key_filters=key_filters,
                                last_modified_filters=last_modified_filters)

        # The scheduling rules. For more information, see the API reference.
        max_schedule_count = 5
        start_cron_expression = "0 0 10 * * ?"
        suspend_cron_expression = "0 0 14 * * ?"
        schedule_rule = ScheduleRule(max_schedule_count=max_schedule_count,
                                     suspend_cron_expression=suspend_cron_expression,
                                     start_cron_expression=start_cron_expression)
        client.create_job(userid, CreateJobRequest(CreateJobInfo(
            name=job_name,
            transfer_mode=transfer_mode,
            overwrite_mode=overwrite_mode,
            src_address=src_address,
            dest_address=dest_address,
            import_qos=import_qos,
            filter_rule=filter_rule,
            schedule_rule=schedule_rule,
        )))
        client.update_job(userid, job_name, UpdateJobRequest(UpdateJobInfo(status="IMPORT_JOB_LAUNCHING")))

        """
        Step 5: Poll the task status.
        """
        while True:
            response = client.get_job(userid, job_name, GetJobRequest())
            if response.body.import_job.status == "IMPORT_JOB_INTERRUPTED":
                print("job is interrupted")
                sys.exit()
            if response.body.import_job.status == "IMPORT_JOB_FINISHED":
                print("job is finished")
                break
            time.sleep(60)

        """
        Step 6: View the results after the task is complete.
        """
        response = client.list_job_history(userid, job_name, ListJobHistoryRequest())
        print(response.body.job_history_list)
    except Exception as e:
        print(e)

Scenario 2: Retry failed files

After a migration task is complete, some files may fail to migrate. The migration service generates a list of failed files that you can use to retry migration:

  1. Poll the task status and wait for completion.

  2. Query the task history to check for failed files.

  3. Wait for the retry result to be ready.

  4. Create a new data address based on the failed file list.

  5. Create and start a retry task.

Note

Retrying failed files through the API is complex. Consider using the console for a simpler experience.

import os
import sys
import time
from datetime import datetime

from alibabacloud_hcs_mgw20240626.client import Client
from alibabacloud_hcs_mgw20240626.models import *
from alibabacloud_tea_openapi.models import Config


def retry(client, userid, job_name, src_address, dest_address, runtime_id):
    """
    Create a data address based on the failed file list and start a retry task.
    """
    result_response = client.get_job_result(userid, job_name, GetJobResultRequest(
        runtime_id=runtime_id
    ))
    job_response = client.get_job(userid, job_name, GetJobRequest())
    address_response = client.get_address(userid, src_address)
    now_time = str(int(datetime.now().timestamp()))
    retry_src_address = src_address + "_retry_" + now_time
    client.create_address(userid, CreateAddressRequest(CreateAddressInfo(
        name=retry_src_address,
        address_detail=AddressDetail(
            address_type=result_response.body.import_job_result.address_type,
            inv_domain=result_response.body.import_job_result.inv_domain,
            inv_location=result_response.body.import_job_result.inv_location,
            inv_bucket=result_response.body.import_job_result.inv_bucket,
            inv_access_id=result_response.body.import_job_result.inv_access_id,
            inv_access_secret=result_response.body.import_job_result.inv_access_secret,
            inv_path=result_response.body.import_job_result.inv_path,
            inv_region_id=result_response.body.import_job_result.inv_region_id,
            domain=address_response.body.import_address.address_detail.domain,
            region_id=address_response.body.import_address.address_detail.region_id,
            access_id=address_response.body.import_address.address_detail.access_id,
            access_secret=address_response.body.import_address.address_detail.access_secret,
            agent_list=address_response.body.import_address.address_detail.agent_list,
            prefix=address_response.body.import_address.address_detail.prefix,
            role=address_response.body.import_address.address_detail.role,
            bucket=address_response.body.import_address.address_detail.bucket
        )
    )))
    verify_response = client.verify_address(userid, retry_src_address)
    if verify_response.body.verify_address_response.status != "available":
        print("verify job failed, status: %s" % verify_response.body.verify_address_response.status)

    # Create and start a retry task.
    retry_job_name = job_name + "_retry_" + now_time
    client.create_job(userid, CreateJobRequest(CreateJobInfo(
        name=retry_job_name,
        transfer_mode=job_response.body.import_job.transfer_mode,
        overwrite_mode=job_response.body.import_job.overwrite_mode,
        create_report=job_response.body.import_job.create_report,
        audit=job_response.body.import_job.audit,
        src_address=retry_src_address,
        dest_address=dest_address,
        parent_version=result_response.body.import_job_result.version
    )))
    client.update_job(userid, retry_job_name, UpdateJobRequest(UpdateJobInfo(
        status="IMPORT_JOB_LAUNCHING"
    )))


if __name__ == "__main__":
    try:
        # The ID of your Alibaba Cloud account.
        userid = "11470***876***55"
        # In this example, the endpoint in the China (Beijing) region is used.
        endpoint = "cn-beijing.mgw.aliyuncs.com"

        """
        Step 1: Initialize a client.
        """
        # Supports HTTPS and HTTP. If not specified, HTTPS is used by default.
        config = Config(
            endpoint=endpoint,
            access_key_id=os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_ID"),
            access_key_secret=os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_SECRET"),
            protocol="http"
        )
        client = Client(config)

        # Specify the following parameters based on your business requirements.
        job_name = "examplejob"
        runtime_id = 1
        src_address = "examplesrcaddress"
        dest_address = "exampledestaddress"

        """
        Step 2: Poll the task status and wait for completion.
        """
        while True:
            response = client.get_job(userid, job_name, GetJobRequest())
            if response.body.import_job.status == "IMPORT_JOB_INTERRUPTED":
                print("job is interrupted")
                sys.exit()
            if response.body.import_job.status == "IMPORT_JOB_FINISHED":
                print("job is finished")
                break
            time.sleep(60)

        """
        Step 3: Query the task history to check for failed files.
        """
        count = 1
        marker = ""
        response = client.list_job_history(userid, job_name, ListJobHistoryRequest(
            count
        ))
        if len(response.body.job_history_list.job_history) < 1:
            print("list job history failed, length < 1")
            sys.exit()
        if int(response.body.job_history_list.job_history[0].failed_count) <= 0:
            print("job %s no need retry" % job_name)
            sys.exit()

        """
        Step 4: Wait for the retry result to be ready.
        """
        while True:
            response = client.get_job_result(userid, job_name, GetJobResultRequest(
                runtime_id=runtime_id
            ))
            if response.body.import_job_result.ready_retry == "NoNeed":
                print("job no need retry")
                sys.exit()
            if response.body.import_job_result.ready_retry == "Ready":
                break
            time.sleep(60)

        retry(client, userid, job_name, src_address, dest_address, runtime_id)
    except Exception as e:
        print(e)

API methods used in this topic

The following table lists the API methods used in the sample code. In the API, a migration task is called a job.

Method

Description

create_tunnel

Creates a tunnel for managing bandwidth and QPS limits.

create_agent

Creates a migration agent associated with a tunnel.

create_address

Creates a source or destination data address.

verify_address

Verifies the availability of a data address.

create_job

Creates a migration task.

update_job

Updates the status of a task. Used to start a task by setting the status to IMPORT_JOB_LAUNCHING.

get_job

Queries the details and status of a task.

list_job_history

Queries the execution history of a task.

get_job_result

Queries the result of a task run. Used for retry operations.

get_address

Queries the details of a data address.