Data migration

Considerations

Consider the following when using the online migration service:

• The online migration service accesses source data using the standard public APIs of the source storage service provider. The migration behavior depends on the provider's service implementation.

• Online migration consumes resources at both the source and the destination, potentially impacting your business operations. If your services are critical, evaluate the potential impact and configure a speed limit for the migration task, or run the task during off-peak hours.

• The online migration service checks files at both the source and destination before a migration begins. However, if a file at the source has the same name as a file at the destination and the migration task is configured to overwrite existing files, the service overwrites the destination file. If the files contain different data, rename one of the files or back up the destination file before starting the migration.

• The online migration service preserves the last modified time of source files. If a lifecycle rule is configured for the destination bucket, a migrated file whose last modified time matches the rule's conditions may be deleted or transitioned to the specified archive storage class.

Limitations

• If static website hosting is enabled for the source bucket, the migration service also detects directory paths as objects. For example, if you upload the file myapp/resource/1.jpg, the migration service detects three objects: myapp/, myapp/resource/, and myapp/resource/1.jpg. Migration for the directory objects myapp/ and myapp/resource/ will fail, but the file myapp/resource/1.jpg will migrate successfully.

• If the source bucket contains symbolic links, the migration service transfers the symbolic links themselves, not the objects they point to. For more information about symbolic links, see symbolic links.

• You can migrate data from only one bucket per migration task. Migrating data from an entire account at once is not supported.

• Data migration from Financial Cloud and Government Cloud is not supported.

• When you migrate data between Alibaba Cloud OSS buckets, object properties are handled as follows:

  • The following properties are migrated: x-oss-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.

  • The following properties are not migrated (including but not limited to): storage class, ACL, server-side encryption, tagging, and user-defined headers such as x-oss-persistent-header.

    Note

    The list of unsupported properties is not exhaustive. The migration behavior of unlisted properties is not guaranteed. Verify your object properties after the migration is complete.

Step 1: Select a region

Step 2: Create a source address

1. In the navigation pane on the left, choose Data Online Migration > Address Management, and then click Create Address.

2. In the Create Address panel, configure the following parameters, and then click OK.

   Parameter	Required	Description
   Name	Yes	The name must be 3 to 63 characters in length. The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive. The name cannot start with a hyphen (-) or an underscore (_).
   Type	Yes	Select OSS.
   Custom domain name	No	You can specify a custom domain name.
   Region	Yes	Select the region where the source bucket is located, for example, China (Hangzhou).
   RAM role	Yes	If the source bucket is in your current account: Automatic RAM role authorization (Recommended). Manual RAM role authorization (Alternative). If the source bucket belongs to a different account: Authorize in the OSS console.
   Bucket	Yes	Enter the name of the source bucket in the current account.
   Prefix	No	You can specify a prefix to migrate a subset of your data. The prefix cannot start with a forward slash (/) but must end with one, for example, data/to/oss/. If you specify a prefix: For example, if the source prefix is example/src/, an object with the key example/src/example.jpg is selected for migration. If you set the destination prefix to example/dest/, the object's key after migration becomes example/dest/example.jpg. If you do not specify a prefix: For example, if you do not specify a source prefix and the object key is srcbucket/example.jpg, and you set the destination prefix to destbucket/, the object's key after migration becomes destbucket/srcbucket/example.jpg.
   Tunnel	No	Select the name of the channel to use. Important This parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system. You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
   Agent	No	Select the name of the agent to use. Important This parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system. You can select a maximum of 200 agents for a specified channel.

Step 3: Create a destination address

1. In the navigation pane on the left, choose Data Online Migration > Address Management, and then click Create Address.

2. In the Create Address panel, configure the following parameters, and click OK.

Parameter	Required	Description
Name	Yes	Enter a name for the destination address. The name must meet the following requirements: The name must be 3 to 63 characters in length. The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive. The name cannot start with a hyphen (-) or an underscore (_).
Type	Yes	Select OSS.
Custom domain name	No	The custom domain name for accessing the bucket.
Region	Yes	Select the region where the destination bucket is located, for example, China (Hangzhou).
Role	Yes	If the destination bucket is in your current Alibaba Cloud account: Authorize the Online Migration Service console (Recommended). Manually authorize the Online Migration Service console (Alternative). If the destination bucket belongs to a different Alibaba Cloud account: Authorize in the OSS console.
Bucket	Yes	The name of the destination bucket.
Prefix	No	Specify a directory (prefix) to place migrated objects in the destination bucket. The prefix cannot start with a forward slash (/) and must end with one, for example, data/to/oss/. If you specify a prefix: For example, if an object named example.jpg is in the source path example/src/, and you set the destination prefix to example/dest/, the object's full path in the destination becomes example/dest/example.jpg. If this parameter is left blank, data is migrated to the root of the destination bucket.
Tunnel	No	Select the name of the channel to use. Important This parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system. You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent	No	Select the name of the agent to use. Important This parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system. You can select a maximum of 200 agents for a specified channel.

Step 4: Create a migration task

1. In the left-side navigation pane, choose Data Online Migration > Migration Tasks, and then click Create Task.

2. On the Select Address page, configure the following parameters, and then click Next.

   Parameter	Required	Description
   Name	Yes	Enter a name for the migration task. The name must meet the following requirements: The name must be 3 to 63 characters in length. The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive. The name cannot start with a hyphen (-) or an underscore (_).
   Source Address	Yes	Select an existing source address.
   Destination Address	Yes	Select an existing destination address.

3. On the Task Configurations page, configure the following parameters.

   Parameter	Required	Description
   Basic settings
   Migration Bandwidth	No	Select the migration bandwidth. Default: The maximum available bandwidth. The actual speed depends on the file size and the number of files. Specify an upper limit: Specify a maximum bandwidth limit. Important The actual migration bandwidth depends on factors such as the data source, network conditions, destination throttling, and file size, and may not reach the specified upper limit. Select a value based on your data source, destination, business requirements, and network bandwidth. Improper throttling can disrupt your business operations.
   Files Migrated Per Second	No	Select the number of files migrated per second. Default: Uses the default rate for files migrated per second. Specify an upper limit: Specify a maximum number of files migrated per second. Important The actual number of files migrated per second is affected by factors such as the data source, network conditions, destination throttling, and file size, and may not reach the specified upper limit. Select a value based on your data source, destination, business requirements, and network bandwidth. Improper throttling can disrupt your business operations.
   Overwrite Mode	Yes	Specifies how to handle files with the same name. Do not overwrite: Skips the file. Overwrite All: The source file overwrites the destination file. Overwrite based on the last modification time: If the last modified time of the source file is later than that of the destination file, the destination file is overwritten. If the last modified time of the source and destination files are the same, the destination file is overwritten if their Size or Content-Type differs. Warning With the Overwrite based on the last modification time policy, there is a risk that an older file may overwrite a newer file because it does not strictly guarantee that newer files will be preserved. If you select the Overwrite based on the last modification time policy, ensure that the source files can return metadata such as last modified time, Size, and Content-Type. Otherwise, the overwrite policy may not work as expected. When you select Do not overwrite or Overwrite if last modified time is later, the service sends one request to the source and one to the destination to retrieve metadata for the overwrite check. This incurs request fees at both the source and the destination.
   Audit settings
   Migration Report	Yes	Specifies how to push the migration report. Do not push (Default): Does not push the migration report to the destination bucket. Push: Pushes the migration report to the destination bucket. For path details, see Next steps. Important Pushing the migration report consumes storage space at the destination. Pushing the migration report may be delayed. Each task execution record has a unique ID. The migration report is pushed only once. Do not delete it unless necessary.
   Migration Logs	Yes	Specifies how to push the migration log. Do not push (Default): Does not push the migration log. Push: Pushes the migration log to Log Service (SLS). You can view the migration log in SLS. Push only file error logs.: Pushes only file error logs to Log Service (SLS). You can view these logs in SLS. When you select Push or Push only file error logs., Online Migration Service creates a project in Log Service (SLS) named aliyun-oss-import-log-<Alibaba Cloud account ID>-<current deployment region>, for example, aliyun-oss-import-log-137918634953****-cn-hangzhou. Important Before you select Push or Push only file error logs., ensure you have completed the following operations. Otherwise, the migration task may fail. You have activated Log Service (SLS). You have granted the required permissions on the Authorize page.
   Authorize	No	This option appears only when you set Migration Logs to Push or Push only file error logs.. Click Authorize to go to the Cloud Resource Access Authorization page. The system creates the AliyunOSSImportSlsAuditRole role and grants it the necessary permissions. Click Agree to Authorization to complete the authorization.
   Filters
   File Name	No	A filter based on filenames. Supports Include and Exclude rules using regular expressions based on the RE2 library (only a subset of the syntax is supported). For example: .*\.jpg$ matches all files that end with .jpg. ^file.* matches all files at the root level that start with file by default. If you set a prefix for the source address, such as data/to/oss/, you must use ^data/to/oss/file.* to match all files that start with file under that prefix. .*/picture/.* matches any subdirectory named picture. Important When you use an Include rule, all matching files are migrated. If there are multiple Include rules, any file that matches at least one rule is migrated. For example, if you have two files, picture.jpg and picture.png, and you set an Include rule for .*\.jpg$, only picture.jpg is migrated. If you also add an Include rule for .*\.png$, both files are migrated. When you use an Exclude rule, no matching files are migrated. If there are multiple Exclude rules, any file that matches at least one rule is not migrated. For example, if you have two files, picture.jpg and picture.png, and you set an Exclude rule for .*\.jpg$, only picture.png is migrated. If you also add an Exclude rule for .*\.png$, neither file is migrated. Exclude rules take precedence. If a file matches both an Exclude rule and an Include rule, the file is not migrated. For example, for a file named file.txt, if you set an Exclude rule for .*\.txt$ and an Include rule for file.*, the file file.txt is not migrated.
   File Modification Time	No	Filters files based on their last modified time. You can specify a time range to migrate only files whose last modified time falls within that range. The rules are as follows: If you specify only a start time of January 1, 2019, the task migrates only files last modified on or after that date. If you specify only an end time of January 1, 2022, the task migrates only files last modified on or before that date. If you specify a start time of January 1, 2019 and an end time of January 1, 2022, the task migrates only files last modified on or between these two dates.
   Migration configuration
   Retain last modified time	Yes	Specifies whether to retain the last modified time of the source file. Retain (Default): The destination file retains the last modified time of the source file. Do not retain: The last modified time is not set.
   Convert Appendable Object to Normal Object/Multipart Object	No	Specifies whether to convert Appendable Objects to Normal Objects or Multipart Objects. Yes: Converts source Appendable Objects to Normal Objects or Multipart Objects at the destination. No: Source Appendable Objects are migrated to the destination as Appendable Objects. Important This option is visible and configurable only if your account is on the allowlist. This option is currently supported only for OSS-to-OSS migration tasks.
   Specify destination storage class	No	Specifies whether to set the storage class for destination files. Specify: Migrated files are assigned the specified storage class at the destination. The following storage classes are available: Standard Infrequent Access Archive Cold Archive Deep Cold Archive Do not specify (Default): Migrated files at the destination use the default storage class of the destination bucket. Important This option is visible and configurable only if your account is on the allowlist. This option is currently supported only for tasks with an OSS destination.
   Task scheduling
   Execution time	No	Important If a migration task is still running at its next scheduled execution time, that execution is postponed until the next scheduled time after the current run completes. This continues until the specified number of executions is reached. Concurrency limit for migration tasks: Deployment regions in the Chinese mainland and China (Hong Kong) support a maximum of 10 concurrent tasks. Overseas regions support a maximum of 5 concurrent tasks. Exceeding this limit may cause scheduled tasks to fail. Defines when to run the migration task. Immediately: Runs the task as soon as it is created. At the Specified Time: Specifies a daily time window for the task to run. By default, the task starts at the specified start time and pauses at the specified stop time. Periodic Scheduling: Runs the task based on a specified frequency and number of executions. Execution frequency: Supports five types of frequencies: Hourly, Daily, Weekly, Specific days of the week, and Custom. For more information, see Execution frequency reference. Number of executions: Specifies how many times the task runs. If not set, the task runs once by default. For the maximum number of executions, refer to the console prompt. Important You can manually start and stop tasks at any time, regardless of the custom execution time settings.

4. Read the Online Migration Service Agreement, select I understand and confirm the compliance commitment statement and acknowledge my responsibility to verify data consistency after the migration task is complete, and then click Next.

5. Review your configuration settings. If everything is correct, click OK and wait for the migration task to run.

Execution frequency

Step 5: Verify data