migration task-Data Online Migration(Data Online Migration )-阿里云帮助中心

A migration task copies data from the source to the destination based on your specified bandwidth, concurrency, overwrite policy, and filter rules. The task also generates migration logs and reports for auditing and troubleshooting.

Basic configuration

Basic configuration for a migration task includes the task name, source address, and destination address.

Task name

The name of the migration task must meet the following requirements:

The name cannot be empty and must be 3 to 63 characters long.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_), and is case-sensitive.
The name cannot start with a hyphen (-) or an underscore (_).

Address configuration

Source Address: The address from which the migration task reads data.
Destination Address: The address to which the migration task writes data.

Throttling

Throttling controls network bandwidth and the file transfer rate during migration to avoid affecting your business workloads. The actual migration bandwidth depends on factors such as the data source, network conditions, and file sizes, and may not reach the specified limit. Evaluate your data source, business requirements, and network bandwidth to select appropriate values. Inappropriate throttling may disrupt your business operations. The maximum migration bandwidth and QPS vary by region.

Migration bandwidth

The maximum network bandwidth that a migration task can use. You can adjust the bandwidth limit at any time during a migration based on your business workload. The change takes some time to take effect.

File transfer rate

The maximum number of files that can be migrated per second. This setting controls file transfer concurrency. You can adjust this limit at any time during a migration based on your needs. The change takes some time to take effect.

Overwrite policy

This policy determines what to do when a file with the same name exists at the destination. A well-chosen overwrite policy can reduce redundant data transfers during iterative migrations, saving bandwidth and time.

Overwrite policy options

Do not overwrite: Skips the file. The file is not migrated.
Overwrite All: The source file overwrites the destination file.
Overwrite based on the last modification time:
- If the source file has a later last-modified time, it overwrites the destination file.
- If the files have the same last-modified time but differ in size or Content-Type, the destination file is overwritten.

Important

The Overwrite based on the last modification time option does not guarantee that a newer file will never be overwritten by an older one.
If you select the Overwrite based on the last modification time policy, you must ensure that the source can return metadata such as last-modified time, size, and Content-Type. Otherwise, the policy might not work as expected and could lead to unintended migration results.
When you select Never overwrite or Overwrite if newer, the system retrieves metadata from both the source and the destination to make the overwrite decision. This action incurs request fees at both endpoints.

Filters

Use filters to precisely control which files to migrate. You can filter by filename patterns and last-modified time.

Note

Filters scan all source files and migrate only those that match your rules. This full scan can take a long time if the source contains a large number of files.
Filename filters apply to the absolute path of each source file.

Filename filter

File Name: Set a filename filter to precisely control which files to migrate.

You can set both Include and Exclude filter rules. The rules use a subset of the RE2 regular expression syntax.

Rule examples

.*\.jpg$ matches all files that end with .jpg.
^file.* matches all files in the root directory that start with "file" by default.
If the source address has a prefix, such as data/to/oss/, you must use ^data/to/oss/file.* to match files that start with "file" under that specific prefix.
.*/picture/.* matches any file within a subdirectory named "picture" at any level of the directory structure.

Filter rules

Include rules: Files that match an include rule are migrated. If you define multiple include rules, any file that matches at least one rule is migrated.
For example, consider two files: picture.jpg and picture.png. If you set an include rule to .*\.jpg$, only picture.jpg is migrated. If you add another include rule for .*\.png$, both files are migrated.
Exclude rules: Files that match an exclude rule are not migrated. If you define multiple exclude rules, any file that matches at least one rule is not migrated.
For example, consider two files: picture.jpg and picture.png. If you set an exclude rule to .*\.jpg$, only picture.png is migrated. If you add another exclude rule for .*\.png$, neither file is migrated.

Priority: Exclude rules take precedence over include rules. If a file matches both an exclude rule and an include rule, the file is not migrated.

For example, for a file named file.txt, if you set an exclude rule to .*\.txt$ and an include rule to file.*, the file is not migrated.

Last-modified time filter

File Modification Time: Filter files based on their last-modified time.

The system migrates only files whose last-modified time falls within the specified range.

Specify only the start time: For example, if you set the start time to "2019-01-01 00:00:00", only files modified on or after this time are migrated.
Specify only the end time: For example, if you set the end time to "2022-01-01 00:00:00", only files modified on or before this time are migrated.
Specify both start and end times: For example, if the start time is "2019-01-01 00:00:00" and the end time is "2022-01-01 00:00:00", all files modified within this time range (inclusive) are migrated.

Task scheduling

Set the start time and execution plan for a migration task. You can run a task immediately, at a specified time, or on a recurring schedule.

Scheduling

If a recurring task is still running at its next scheduled start time, the current run finishes, and the next scheduled run is skipped.
If the number of concurrent migration tasks exceeds the limit, scheduled tasks might not run as expected.

Execution modes

Immediately: Starts the task right away.
At the Specified Time: Specify a daily time window for the task. By default, the task starts at the specified start time and pauses at the specified stop time.
Periodic Scheduling: Configure the task by setting the execution frequency and the maximum number of runs.
- Execution frequency: You can choose from five frequency options: Hourly, Daily, Weekly, Specific days of the week, or Custom. For more information, see Execution frequency details.
- Maximum number of runs: A single run is one complete execution of the task from start to finish. The task is automatically scheduled up to this limit. Once the limit is reached, automatic scheduling stops, but you can still run the task manually. For the limit on the maximum number of runs, refer to the prompt in the console.

You can manually start or pause a task at any time, regardless of its scheduled execution time.

Execution frequency

The following table describes the execution frequency options.

Execution frequency	Description	Example
Hourly	Run the task once every hour. You can use this option with the maximum number of runs.	The current time is 8:05. The frequency is set to hourly with a maximum of 3 runs. The first run starts at the next hour, 9:00. If a run finishes before the next hour, the second run starts at 10:00. This pattern continues until the specified number of runs is complete. If a run has not finished by the next hour and ends at 12:30, the second run starts at the next hour, 13:00. This pattern continues until the specified number of runs is complete.
Daily	Run the task once a day. You must specify an hour (0-23) for the task to start. You can use this option with the maximum number of runs.	The current time is 8:05. The task is scheduled to run daily at 10:00, with a maximum of 5 runs. The first run starts at 10:00 today. If a run finishes before 10:00 the next day, the second run starts at 10:00 the next day. This pattern continues until the specified number of runs is complete. If a run has not finished by 10:00 the next day and ends at 12:05 the next day, the second run starts at 10:00 on the third day. This pattern continues until the specified number of runs is complete.
Weekly	Run the task once a week. You must specify a day of the week and an hour (0-23) for the task to start. You can use this option with the maximum number of runs.	The current time is Monday, 8:05. The task is scheduled to run every Monday at 10:00, with a maximum of 10 runs. The first run starts at 10:00 today. If a run finishes before 10:00 next Monday, the second run starts at 10:00 next Monday. This pattern continues until the specified number of runs is complete. If a run has not finished by 10:00 next Monday and ends at 12:05 next Monday, the second run starts at 10:00 on the following Monday. This pattern continues until the specified number of runs is complete.
Specific days of the week	Run the task on selected days of the week. You must specify the days and an hour (0-23) for the task to start.	The current time is Wednesday, 8:05. The task is scheduled to run on Mondays, Wednesdays, and Fridays at 10:00. The first run starts at 10:00 today. If a run finishes before 10:00 on Friday, the second run starts at 10:00 on Friday. This pattern continues until the specified number of runs is complete. If a run has not finished by 10:00 on Friday and ends at 12:05 next Monday, the second run starts at 10:00 next Wednesday. This pattern continues until the specified number of runs is complete.
Custom	Use a cron expression to define a custom schedule for the task start time.	Note A cron expression consists of six space-separated fields that define the execution schedule: second, minute, hour, day of the month, month, and day of the week. The minimum interval is 1 hour. The following cron expression examples are for reference only. For more options, use a cron expression generator. `0 0 * * * `: Runs the task at the beginning of every hour (0 minutes, 0 seconds). `0 30 0/3 * ?`: Runs the task every 3 hours at 30 minutes past the hour (for example, at 0:30, 3:30, 6:30, 9:30, 12:30, 15:30, 18:30, and 21:30). `0 0 12 * * MON-FRI`: Runs the task at 12:00 PM every weekday from Monday to Friday. `0 0 12 1-15 * SAT,SUN`: Runs the task at 12:00 PM on weekends (Saturday and Sunday) that fall between the 1st and 15th of the month. `0 30 8 1,15 * *`: Runs the task at 8:30 AM on the 1st and 15th of each month.

Migration reports

A migration report provides detailed results, including lists of successfully migrated, failed, and skipped files, to help you understand the execution status of a migration task.

Report file types

A migration task generates three types of report files: a list of all migrated files, a list of failed files, and a list of skipped files.

Report file naming convention

uid@jobid@runtimeid_total_list_n: Represents the list of all migrated files. Multiple files may be generated, where n is an integer equal to or greater than 0.
uid@jobid@runtimeid_failed_list_n: Represents the list of failed files. Multiple files may be generated, where n is an integer equal to or greater than 0.
uid@jobid@runtimeid_skipped_list_n: Represents the list of skipped files. Multiple files may be generated, where n is an integer equal to or greater than 0.

Report fields

The fields in a migration report describe the properties of each migrated file.

Field	Description
Source filename	The URL-encoded name of the source file.
Destination filename	The URL-encoded name of the destination file.
Source file size	The size of the source file.
Destination file size	The size of the destination file.
Source file MD5	The MD5 hash of the source file. This field is present only if the source returns an MD5 hash.
Destination file MD5	The MD5 hash of the destination file.
Source file CRC64	The CRC64 hash of the source file. This field is present only if the source returns a CRC64 hash.
Destination file CRC64	The CRC64 hash of the destination file.
Source file last-modified time	The last-modified time of the source file.
Destination file last-modified time	The last-modified time of the destination file.
Source object version ID (multi-version migration only)	The version ID of the source object. This field is used only in multi-version migrations.
Destination object version ID (multi-version migration only)	The version ID of the destination object. This field is used only in multi-version migrations.
Migration start time	The time when the file migration started.
Migration end time	The time when the file migration ended.
File migration status	`transferred`: The file migration was successful. `failed`: The file migration failed. `skipped`: The file migration was skipped.
Exception reason	The reason for the exception.

Retrieve a migration report

Log on to the Data Online Migration console.
In the left-side navigation pane, choose Data Online Migration > Migration Tasks.

In the list of migration tasks, find the task that you want to manage and click Manage to open the task details page.

If you selected Do not push reports when you created the task, go to the History tab and click Generate Migration Report. After the report is generated, you can click Download Migration Report to save it locally.
If you selected Push reports when you created the task, go to the History tab. After the report is generated, you can click Download Migration Report to save it locally.

Note

A migration report generated before the migration task is complete may be incomplete. To get a complete report, download it after the task has finished.
Migration reports are for reference only. The actual content at the destination is the definitive source for the migration status.

After a migration report is generated, it is stored in the specified Object Storage Service (OSS) bucket. The directory prefix format varies based on the task execution status, as shown below:

OSS://<bucket>/<prefix>/aliyun_import_report/<uid>/<jobid>/<runtimeid>/total_list/
OSS://<bucket>/<prefix>/aliyun_import_report/<uid>/<jobid>/<runtimeid>/failed_list/
OSS://<bucket>/<prefix>/aliyun_import_report/<uid>/<jobid>/<runtimeid>/skipped_list/

Path parameters

Parameter	Description
bucket	The destination bucket.
prefix	The directory prefix specified in the destination address.
uid	Your Alibaba Cloud account ID.
jobid	The task ID. You can find the ID on the Details tab of the task details page.
runtimeid	The execution record ID. You can find the ID on the History tab of the task details page.

In the OSS console, locate the migration report files in the destination bucket. You can download these files to view the detailed file lists. We recommend using ossbrowser 2.0 (Preview) or the ossutil tool to view the files.

Migration logs

Migration logs can be pushed to Simple Log Service (SLS) to record the handling of each file in a migration task, including success, failure, and skipped events. With migration logs, you can monitor migration progress in real time, quickly identify problematic files, and perform audits and troubleshooting.

Log types

Data Online Migration creates a project in SLS. The project is named in the format aliyun-oss-import-log-<Alibaba-Cloud-account-ID>-<region-ID>, for example, aliyun-oss-import-log-137918634953****-cn-hangzhou. In this project, you can view the logs for your migration task, which include the following two types:

drs_import_success_log: Logs for successful migrations.
drs_import_fail_log: Logs for failed migrations.

Log fields

The following table describes the main fields in the migration logs.

Field	Description
JobName	The name of the migration task.
JobId	The ID of the migration task.
Region	The region where the console is located.
Status	The migration status of the file. Valid values: succeed: The migration was successful. skipped: The file was skipped. A file is skipped in the following scenarios: The source file does not exist at the time of migration (it has been deleted). The file was skipped based on the configured overwrite policy. failed: The migration failed.
UserId	Your Alibaba Cloud account ID.
ExecuteId	The execution record ID.
StartTime	The time when the migration started.
EndTime	The time when the migration ended.
ListTime	The time when the file was scanned.
Time	The current time.
SrcObjectName	The name of the source file. Fields that start with Src are related to the source. The filename is URL-encoded, for example, `docs%2Fmy.docir%2Fexample.log`.
SrcObjectSize	The size of the source file.
DestObjectName	The name of the destination file. Fields that start with Dest are related to the destination. The filename is URL-encoded, for example, `docs%2Fmy.docir%2Fexample.log`.
DestObjectSize	The size of the destination file.

Retrieve migration logs

You can push migration logs to SLS in real time for querying and analysis. In the SLS console, you can search by task ID, filename, or other keywords to monitor the migration process or troubleshoot issues.

Important

To use the log push feature, you must enable it when you create the migration task by selecting Push or Push only file error logs. If you select Do not push during task creation, the logs for this task are not available in SLS.
Migration logs are not guaranteed to be complete. Do not use them for migration integrity checks.
Migration logs are for reference only. The actual files at the destination are the definitive source for the migration status.

Do not push (Default): Does not push migration logs.
Push: Pushes migration logs to SLS, where you can view them.
Push only file error logs.: Pushes only error logs to SLS, where you can view them.
Before you select Push or Push only file error logs, make sure you have completed the following prerequisites. Otherwise, the migration task may fail:
- You have activated SLS.
- You have granted authorization on the Authorize page.

When you select Push or Push only file error logs., Data Online Migration creates a project in SLS. The project is named in the format aliyun-oss-import-log-<Alibaba-Cloud-account-ID>-<region-ID-of-the-migration-service>, for example, aliyun-oss-import-log-137918634953****-cn-hangzhou.

Authorize: This option appears if you select Push or Push only file error logs for the Migration log setting.
Click Authorize to go to the Resource Access Authorization page. The system creates the AliyunOSSImportSlsAuditRole role and grants it the necessary permissions. Click Agree to Authorization to complete the process.

Migration task management

Task statuses

Created: The task is created. You can start it manually or wait for it to start at the scheduled time.
Starting: The task is created and has been started. It is waiting to be scheduled.
Preparing: Data is being preprocessed.
Migrating: Data migration is in progress. Please wait.
Deleting: The task is being deleted. After the deletion is complete, the task is removed from the task list.
Paused: The task is paused.
Stopping: The task is being stopped. After it stops, its status changes to Finished.
Finished: The task has finished. This status does not guarantee that all files were migrated successfully. Check the migration report for details.
Interrupted: The task was interrupted due to an error.

Retry failed files

If a migration task finishes with failed files, you can create a subtask to retry the failed files.

Procedure

Log on to the Data Online Migration console.
In the top navigation bar, select the region of the task.
In the left-side navigation pane, choose Data Online Migration > Migration Tasks.
In the list of migration tasks, find the task and click Manage in the Actions column.
On the History tab, find the execution record and click Retry Failed Files in the Actions column.
(Optional) Modify the name of the subtask and click Next.
Review the parameters, such as migration bandwidth, file transfer rate, and overwrite policy. Then, click Next.
Click OK. The subtask starts and retries migrating the failed files.