Implementation
This topic describes the usage notes, migration limitations, and procedure for migrating data from OSS, OSS-HDFS, or a third-party source to Alibaba Cloud Object Storage Service (OSS).
Usage notes
The online migration service accesses source data using the standard public APIs of the source storage service provider. The migration behavior depends on the provider's service implementation.
Online migration consumes resources at both the source and the destination, potentially impacting your business operations. If your services are critical, evaluate the potential impact and configure a speed limit for the migration task, or run the task during off-peak hours.
The online migration service checks files at both the source and destination before a migration begins. However, if a file at the source has the same name as a file at the destination and the migration task is configured to overwrite existing files, the service overwrites the destination file. If the files contain different data, rename one of the files or back up the destination file before starting the migration.
The online migration service preserves the last modified time of source files. If a lifecycle rule is configured for the destination bucket, a migrated file whose last modified time matches the rule's conditions may be deleted or transitioned to the specified archive storage class.
Migration limitations
OSS/OSS Inventory
If static website hosting is enabled for the source address, the migration scans additional directories that do not actually exist. For example, after you upload the file myapp/resource/1.jpg and enable static website hosting, the migration service scans three objects: myapp/, myapp/resource/, and myapp/resource/1.jpg. Of these objects, myapp/ and myapp/resource/ fail to migrate, but this does not affect the successful migration of myapp/resource/1.jpg.
If the source address contains symbolic links, the symbolic links are migrated directly during data migration. For more information about symbolic links, see Symbolic links.
Data Online Migration can migrate only one bucket at a time. You cannot migrate the data of an entire account in a single migration.
Data migration is not supported in Finance Cloud or Government Cloud.
The following attributes are migrated for Alibaba OSS and Alibaba OSS Inventory data sources:
Attributes that can be migrated: x-oss-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.
Attributes that cannot be migrated (including but not limited to): StorageClass, ACL, server-side encryption, tagging, and user-defined headers such as x-oss-persistent-header.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
S3/S3 Inventory
You can migrate data from only one bucket per migration task. Migrating data from an entire account at once is not supported.
The following attributes are migrated for AWS S3 and AWS S3 Inventory data sources:
Attributes that can be migrated: x-amz-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.
Attributes that cannot be migrated (including but not limited to): StorageClass, ACL, server-side encryption, and tagging.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
For data sources that are not compatible with the S3 API, such as Volcengine TOS and Netease Cloud, migration may fail and cause unpredictable issues. We recommend that you migrate data over HTTP.
Compatible S3
You can migrate data from only one bucket per migration task. Migrating data from an entire account at once is not supported.
The following attributes are migrated for Compatible S3 data sources:
Attributes that can be migrated: LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.
Attributes that cannot be migrated (including but not limited to): StorageClass, ACL, server-side encryption, and tagging.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
Inventory
You can migrate data from only one bucket per migration task. Migrating data from an entire account at once is not supported.
The following attributes are migrated for Inventory data sources:
The attributes that can be migrated depend on the data type, such as OSS or S3. For more information, see the migration tutorial for the corresponding data source.
Attributes that cannot be migrated: not yet determined. The actual migration result is authoritative.
HTTP/HTTPS
The following attributes are migrated for HTTP/HTTPS data sources:
Attributes that can be migrated: LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.
Attributes that cannot be migrated: not yet determined. The actual migration result is authoritative.
Tencent COS
You can migrate data from only one bucket per migration task. Migrating data from an entire account at once is not supported.
The following attributes are migrated for Tencent COS data sources:
Attributes that can be migrated: x-cos-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.
Attributes that cannot be migrated (including but not limited to): StorageClass, ACL, server-side encryption, and tagging.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
WarningIf the source data address uses a Tencent standard domain name, calling
GetObjectforcibly setsContent-Dispositiontoattachment, and this attribute is migrated to the destination, which causes an inconsistency with the source. When a browser accesses such files (such as HTML and PNG files), a forced download is triggered. To avoid this issue, bind a custom domain name on the Tencent COS website and use that domain name for the migration to preserve the originalContent-Dispositionsetting.
Buckets for which metadata acceleration is enabled cannot be migrated.
Qiniu Kodo
You can migrate data from only one bucket per migration task. Migrating data from an entire account at once is not supported.
The following attributes are migrated for Qiniu Kodo data sources:
Attributes that can be migrated: x-qn-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.
Attributes that cannot be migrated (including but not limited to): StorageClass, ACL, server-side encryption, and tagging.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
Huawei OBS/OBSFS
You can migrate data from only one bucket per migration task. Migrating data from an entire account at once is not supported.
The following attributes are migrated for Huawei OBS data sources:
Attributes that can be migrated: x-obs-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Language, and Expires.
Attributes that cannot be migrated (including but not limited to): StorageClass, Content-Disposition, ACL, server-side encryption, and tagging.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
- Warning
If the source data address uses a Huawei OBS standard domain name, calling
GetObjectforcibly setsContent-Dispositiontoattachment, and this attribute is migrated to the destination, which causes an inconsistency with the source. When a browser accesses such files (such as HTML and PNG files), a forced download is triggered.
Volcengine TOS
You can migrate data from only one bucket per migration task. Migrating data from an entire account at once is not supported.
The following attributes are migrated for Volcengine TOS data sources:
Attributes that can be migrated: x-tos-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.
Attributes that cannot be migrated (including but not limited to): StorageClass, ACL, server-side encryption, and tagging.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
If Volcengine TOS is not compatible with the AWS S3 API, migration may fail and cause unpredictable issues. We recommend that you migrate data over HTTP.
Buckets for which hierarchical namespace (HNS) is enabled cannot be migrated.
UCloud
Data Online Migration can migrate only one bucket at a time. You cannot migrate the data of an entire account in a single migration.
The following attributes are migrated for UCloud US3 data sources:
Attributes that can be migrated: x-ufile-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.
Attributes that cannot be migrated (including but not limited to): StorageClass, ACL, server-side encryption, and tagging.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
Baidu BOS
You can migrate data from only one bucket per migration task. Migrating data from an entire account at once is not supported.
The following attributes are migrated for Baidu BOS data sources:
Attributes that can be migrated: x-bce-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, and Content-Language.
Attributes that cannot be migrated (including but not limited to): Expires, StorageClass, ACL, server-side encryption, and tagging.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
WarningIf the source data address uses a Baidu BOS standard domain name, calling
GetObjectforcibly setsContent-Dispositiontoattachment, and this attribute is migrated to the destination, which causes an inconsistency with the source. When a browser accesses such files (such as HTML files), a forced download is triggered.
Kingsoft KS3
Data Online Migration can migrate only one bucket at a time. You cannot migrate the data of an entire account in a single migration.
ImportantKingsoft KS3 migrates data by using the AWS S3 protocol.
The following attributes are migrated for Kingsoft KS3 data sources:
Attributes that can be migrated: x-amz-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.
Attributes that cannot be migrated (including but not limited to): StorageClass, ACL, server-side encryption, and tagging.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
Google Cloud Storage
You can migrate data from only one bucket per migration task. Migrating data from an entire account at once is not supported.
The following attributes are migrated for Google Cloud Storage data sources:
Attributes that can be migrated: x-amz-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, and Content-Language.
Attributes that cannot be migrated (including but not limited to): Expires, StorageClass, ACL, server-side encryption, and tagging.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
Youpai USS
You can migrate data from only one bucket per migration task. Migrating data from an entire account at once is not supported.
Youpai USS migrates data by using the S3 protocol standard. For more information, see AWS S3 Compatibility
The following attributes are migrated for Youpai USS data sources:
Attributes that can be migrated: x-amz-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.
Attributes that cannot be migrated (including but not limited to): StorageClass, ACL, server-side encryption, and tagging.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
Azure Blob
Data Online Migration can migrate only one bucket at a time. You cannot migrate the data of an entire account in a single migration.
The following attributes are migrated for Azure Blob data sources:
Attributes that can be migrated: x-ms-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.
Attributes that cannot be migrated (including but not limited to): StorageClass, ACL, server-side encryption, and tagging.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
LocalFS
Empty directories, symbolic links to files or directories, character device files, block device files, socket files, and pipe files at the source data address are not migrated.
If hard links exist at the source data address, the linked files are migrated as regular files.
The attributes of the parent directories of files cannot be migrated.
File permissions such as SUID, SGID, and SBID cannot be migrated.
The attribute limits for migrating LocalFS data to Alibaba Cloud OSS are as follows:
Attributes that can be migrated: ModifyTime -> X-Oss-Meta-Mtime (OSS), Permissions -> X-Oss-Meta-Perms (OSS), and Uid:Gid -> X-Oss-Meta-Owner (OSS)
NotePermissions: the nine permissions, including read, write, and execute.
Uid:Gid: the user ID and the ID of the group to which the user belongs, separated by a colon (:).
Attributes that cannot be migrated (including but not limited to): AccessTime, ChangeTime, Attr, and ACL.
NoteUnsupported attributes include, but are not limited to, those listed above. The behavior of unlisted attributes is undefined., the actual migration result is authoritative.
OSSHDFS
Data Online Migration can migrate only one bucket at a time. You cannot migrate the data of an entire account in a single migration.
Only some attributes of OSS-HDFS can be migrated to the custom meta of the destination OSS address. The following attributes are supported:
Mtime (last modified time).
Owner and Group.
Permission (read, write, and execute).
For attributes and behaviors that are not described in this topic, the actual migration result is authoritative.
FTP
Data Online Migration can migrate only the data in a single FTP directory at a time. You cannot migrate the data of an entire account in a single migration.
Empty directories, hidden files, symbolic links to files or directories, character device files, block device files, socket files, and pipe files at the source data address are not migrated.
Files whose names contain special characters at the source address may fail to migrate.
If a single file exceeds 5 GB and the destination data address type is Alibaba Cloud OSS, the file fails to migrate.
Step 1: Select a region
Log on to the Data Online Migration console.
Region selection: the region in the upper-left corner of the console indicates the region in which the migration service is deployed. For best performance, select the region where the data source resides or the region that is closest to the data source.
If your data source resides outside the Chinese mainland, you must select a migration service that is deployed in a region outside the Chinese mainland and is geographically close to the data source.
Step 2: Create a source address
For LocalFS and FTP sources, you must create a tunnel and an agent before you create the source address. For more information, see Manage tunnels and Agent management.
OSS
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
In the Create Address panel, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select Alibaba OSS.
Custom Domain Name
No
Specifies whether to use a custom domain name.
Region
Yes
Select the region where the source address resides. For example, select China (Hangzhou).
Authorize Role
Yes
The source bucket belongs to the account that is used to log on to the Data Online Migration console
The source bucket does not belong to the account that is used to log on to the Data Online Migration console
Bucket
Yes
Enter the name of the bucket that stores the data to be migrated in the current account.
Prefix
No
You can specify a data path prefix to migrate part of the data. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example:
data/to/oss/.With a prefix specified: For example, assume that the prefix of the source data address is set to
example/src/and this path contains the file example.jpg. If the prefix of the destination data address is set toexample/dest/, the full path of the file example.jpg after migration isexample/dest/example.jpg.Without a prefix specified: For example, assume that no prefix is set for the source data address and the path of the file to migrate is
srcbucket/example.jpg. If the prefix of the destination data address is set todestbucket/, the full path of the file after migration isdestbucket/srcbucket/example.jpg.
Tunnel
No
Select the name of the tunnel that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from self-managed storage to the cloud.
You must associate an agent when the destination data address is LocalFS, or when the migration must go over an Express Connect circuit (such as in Finance Cloud or Apsara Stack).
Agent
No
Select the name of the agent that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from self-managed storage to the cloud.
You can select up to 200 agents for a specified tunnel.
OSS Inventory
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
In the Create Address panel, configure the following parameters, and click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select Alibaba OSS Inventory.
Region
Yes
Select the region where the source address resides. For example, select China (Hangzhou).
Authorize Role
Yes
The source bucket belongs to the account that is used to log on to the Data Online Migration console
The source bucket does not belong to the account that is used to log on to the Data Online Migration console
Bucket
Yes
Enter the name of the bucket that stores the data to be migrated in the current account.
Region
Yes
When Inventory Location is set to Alibaba OSS, enter the region where the OSS inventory resides.
Inventory Authorize Role
Yes
The inventory bucket belongs to the account that is used to log on to the Data Online Migration console
The inventory bucket does not belong to the account that is used to log on to the Data Online Migration console
Bucket
Yes
Enter the name of the bucket where the inventory to be migrated resides. The bucket belongs to the account that is used to log on to the current console.
Inventory Path
Yes
Enter the path of the manifest.json file, such as dir/manifest.json.
Tunnel
No
Select the name of the tunnel that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from self-managed storage to the cloud.
You must associate an agent when the destination data address is LocalFS, or when the migration must go over an Express Connect circuit (such as in Finance Cloud or Apsara Stack).
Agent
No
Select the name of the agent that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from self-managed storage to the cloud.
You can select up to 200 agents for a specified tunnel.
S3
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
On the Create Data Address page, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select AWS S3.
Domain Name
Yes
Enter the endpoint domain name of AWS S3.For more information, see Amazon S3 endpoints.
AccessKeyId
Yes
Enter the AccessKey pair of the IAM user that you created. AWS S3 uses the AccessKey pair to identify the user and verify whether the user has the permissions to read the source data.
SecretAccessKey
Yes
Bucket
Yes
Enter the name of the AWS S3 bucket that stores the data to be migrated.
NoteThe bucket name cannot start or end with invalid characters such as spaces, line breaks, or tabs.
Prefix
No
You can specify a data path prefix to migrate part of the data.
Specify a prefix: migrates the data in the specified directory (prefix). The prefix cannot start with a forward slash (/) and must end with a forward slash (/).
Do not specify a prefix: migrates all data in the bucket.
Tunnel
No
Select the name of the tunnel that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from self-managed storage to the cloud.
You must associate an agent when the destination data address is LocalFS, or when the migration must go over an Express Connect circuit (such as in Finance Cloud or Apsara Stack).
Agent
No
Select the name of the agent that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from self-managed storage to the cloud.
You can select up to 200 agents for a specified tunnel.
S3 Inventory
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
In the Create Data Address panel, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select AWS S3 Inventory.
Domain Name
Yes
Enter the AWS S3 service endpoint. Endpoints are listed in Amazon S3 endpoints.
AccessKeyId
Yes
Enter the IAM user access key. This key authenticates read access to the source data.
SecretAccessKey
Yes
Bucket
Yes
Enter the name of the source AWS S3 bucket.
NoteThe bucket name cannot start or end with spaces, line breaks, or tabs.
Inventory Location
Yes
Select the storage where the inventory resides, and then select AWS S3.
Inventory Domain Name
Yes
When Inventory Location is set to AWS S3, enter the endpoint domain name of the AWS S3 inventory. Endpoints are listed in Amazon S3 endpoints.
Inventory Bucket
Yes
Enter the name of the bucket where the inventory list of the data to be migrated resides.
Inventory Path
Yes
When Inventory Location is set to AWS S3, enter the path of the AWS S3 inventory manifest.json file, such as dir/manifest.json.
Inventory AccessKeyId
Yes
When Inventory Location is set to AWS S3, enter the AccessKey pair that is used to access the AWS S3 inventory. The AccessKey pair is deleted after the migration is complete.
Inventory SecretAccessKey
Yes
Tunnel
No
Select the name of the tunnel that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from self-managed storage to the cloud.
You must associate an agent when the destination data address is LocalFS, or when the migration must go over an Express Connect circuit (such as in Finance Cloud or Apsara Stack).
Agent
No
Select the name of the tunnel that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from self-managed storage to the cloud.
You must associate an agent when the destination data address is LocalFS, or when the migration must go over an Express Connect circuit (such as in Finance Cloud or Apsara Stack).
Compatible S3
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
On the Create Data Address page, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select Compatible S3.
Domain Name
Yes
Enter the endpoint domain name of the data source that is compatible with the S3 protocol.
Region
Yes
If you use a non-standard S3 domain name, enter the region that corresponds to the actual domain name of the source.
AccessKeyId
Yes
Enter the AccessKey pair. The data source uses the AccessKey pair to identify the user and verify whether the user has the permissions to read the source data.
SecretAccessKey
Yes
Bucket
Yes
Enter the name of the bucket that stores the data to be migrated.
NoteThe bucket name cannot start or end with invalid characters such as spaces, line breaks, or tabs.
Prefix
No
You can specify a data path prefix to migrate part of the data.
Specify a prefix: migrates the data in the specified directory (prefix). The prefix cannot start with a forward slash (/) and must end with a forward slash (/).
Do not specify a prefix: migrates all data in the bucket.
Tunnel
No
Select the name of the tunnel that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from self-managed storage to the cloud.
You must associate an agent when the destination data address is LocalFS, or when the migration must go over an Express Connect circuit (such as in Finance Cloud or Apsara Stack).
Agent
No
Select the name of the agent that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from self-managed storage to the cloud.
You can select up to 200 agents for a specified tunnel.
Inventory
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
In the Create Address panel, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select Inventory.
Data Type
Yes
Select the source that you want to migrate based on your business requirements.
Domain Name
Depends on the Data Type
The endpoint address of the source storage service (for example, the endpoint domain name of AWS S3).
Region
Yes (Data Type is set to Alibaba OSS)
Select the region where the source address resides. For example, select China (Hangzhou).
Authorize Role
Yes (Data Type is set to Alibaba OSS)
The source bucket belongs to the account that is used to log on to the Data Online Migration console
The source bucket does not belong to the account that is used to log on to the Data Online Migration console
AccessKey pair
Yes
Enter the AccessKey pair of the account to which the source data belongs (including AccessKeyId and SecretAccessKey), which is used to identify the user and verify whether the user has the permissions to read the source data.
Storage
Yes
Enter the name of the Bucket.
Prefix
No
You can specify a data path prefix to migrate the source data to a specified directory. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example,
data/to/oss/.Specify a prefix: for example, if the prefix of the source data address is
example/src/and this path contains the file example.jpg. If the prefix of the destination data address is set toexample/dest/, the full path of the file example.jpg after migration isexample/dest/example.jpg.Do not specify a prefix: if you do not set a data path prefix, the source data is migrated to the root directory of the destination bucket.
Inventory Location
Yes
The storage where the inventory file resides. You can select Alibaba OSS or Third-party source.
Inventory Path
Yes
Enter the path of the manifest.json file, such as dir/manifest.json.
Inventory Domain Name
Yes (Inventory Location is set to a value other than Alibaba OSS)
When Inventory Location is set to Third-party source, enter the endpoint domain name of the inventory.
InventoryRegion
Depends on the Data Type
If you set the Inventory Location parameter to Alibaba OSS, specify the region in which the OSS inventory list resides.
Authorize Role
Yes (Inventory Location is set to Alibaba OSS)
The inventory bucket belongs to the account that is used to log on to the Data Online Migration console
The inventory bucket does not belong to the account that is used to log on to the Data Online Migration console
Inventory Bucket
Yes
Enter the name of the Bucket where the inventory to be migrated resides. The bucket belongs to the account that is used to log on to the current console.
Inventory AccessKey pair
Yes (Inventory Location is set to a value other than Alibaba OSS)
When Inventory Location is set to a value other than Alibaba OSS , enter the AccessKey pair that is used to access the source inventory (including AccessKeyId and SecretAccessKey), which is used to identify the user and verify whether the user has the permissions to read the inventory files.
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
HTTP/HTTPS
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
In the Create Address panel, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select HTTP/HTTPS.
Protocol
Yes
Select HTTP or HTTPS.
Port
Yes
Enter the port number. The default port is 80 for HTTP and 443 for HTTPS. Valid values: 0 to 65535.
Inventory Location
Yes
The storage where the inventory file resides. You can select Alibaba OSS or AWS S3.
Inventory Path
Yes
Enter the path of the manifest.json file, such as dir/manifest.json.
Inventory Domain Name or Region
Yes
If you set the Inventory Location parameter to Alibaba OSS, specify the region in which the OSS inventory list resides.
When Inventory Location is set to AWS S3, enter the endpoint domain name of the AWS S3 inventory. Endpoints are listed in Amazon S3 endpoints.
Authorize Role
Yes (Inventory Location is set to Alibaba OSS)
The inventory bucket belongs to the account that is used to log on to the Data Online Migration console
The inventory bucket does not belong to the account that is used to log on to the Data Online Migration console
Bucket
Yes
Enter the name of the bucket where the inventory to be migrated resides. The bucket belongs to the account that is used to log on to the current console.
Inventory AccessKeyId
Yes (Inventory Location is set to AW3 S3)
If Inventory Location is set to AWS S3, enter the access key used to access the AWS S3 inventory list. For security, delete this key after the migration is complete.
Inventory SecretAccessKey
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
Tencent COS
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
On the Create Address page, configure the following parameters, and click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select Tencent COS.
Domain Name
Yes
Enter the endpoint of the bucket, such as cos.ap-nanjing.myqcloud.com. For more information about Tencent COS domain names, see Regions and Endpoints.
AccessKeyId
Yes
Enter the cloud API key that you created. Tencent COS uses the key to identify the user and verify whether the user has the permissions to read the source data.
SecretAccessKey
Yes
Bucket
Yes
Enter the bucket name of Tencent COS. A COS bucket name consists of
Custom name-APPID. For example,example-1234567890.NoteIn the Tencent Cloud console, you can view the APPID on the Account Information page.
Prefix
Yes
Specify a prefix to migrate a subset of the data.
-
If specified, only data under that prefix is migrated. The prefix must not start with a forward slash (/) but must end with one.
-
If you omit this parameter, all data in the bucket is migrated.
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
Qiniu Kodo
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
In the Create Address panel, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select Qiniu Kodo.
Domain Name
Yes
Enter the domain name that corresponds to the region where the bucket resides.
You can view the domain name in the Qiniu Kodo console. For more information, see Domain Name Management.
ImportantThe Qiniu Kodo test domain has limits on the number of accesses from a single IP address and on the bandwidth. We recommend that you use a custom origin domain name to create the migration task.
For more information about domain names, see Test Domain Usage Guidelines (The linked document is for reference only and may be outdated due to changes on the source website).
AccessKeyId
Yes
Enter the Qiniu Kodo access key. Qiniu Kodo uses the access key to identify the user and verify whether the user has the permissions to read the source data.
SecretAccessKey
Yes
Bucket
Yes
For a Qiniu object storage bucket, enter only the custom name. For example, if the bucket name is
example-1234567890, enterexample.Prefix
Yes
Specify a prefix to migrate a subset of the data.
-
If specified, only data under that prefix is migrated. The prefix must not start with a forward slash (/) but must end with one.
-
If you omit this parameter, all data in the bucket is migrated.
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
Huawei OBS/OBSFS
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
On the Create Data Address page, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
For standard object storage, select Huawei OBS.
For a parallel file system, select Huawei OBSFS.
Domain Name
Yes
Enter the endpoint domain name of Huawei OBS.The endpoint is in the following format:
obs.<Region where the bucket resides>.myhuaweicloud.com. For example, if the bucket resides in the CN North-Beijing1 region, the endpoint isobs.cn-north-1.myhuaweicloud.com. For more information about the regions that Huawei OBS supports, see Huawei OBS.AccessKeyId
Yes
Enter the AccessKey pair of the IAM user that you created. Huawei OBS uses the AccessKey pair to identify the user and verify whether the user has the permissions to read the source data.
SecretAccessKey
Yes
Bucket
Yes
Enter the name of the Huawei OBS object storage or parallel file system bucket where the data to be migrated resides.
NoteThe bucket name cannot start or end with invalid characters such as spaces, line breaks, or tabs.
Prefix
No
You can specify a data path prefix to migrate part of the data.
Specify a prefix: migrates the data in the specified directory (prefix). The prefix cannot start with a forward slash (/) and must end with a forward slash (/).
Do not specify a prefix: migrates all data in the bucket.
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
Volcengine TOS
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
On the Create Data Address page, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select Volcengine TOS.
Domain Name
Yes
Enter the endpoint domain name of Volcengine TOS.For more information, see Regions and endpoints supported by Volcengine TOS.
NoteFor the region and domain name, select the corresponding value from the Volcengine TOS table: S3 Endpoint.
AccessKeyId
Yes
Enter the AccessKey pair of the IAM user that you created. Volcengine TOS uses the AccessKey pair to identify the user and verify whether the user has the permissions to read the source data.
SecretAccessKey
Yes
Bucket
Yes
Enter the name of the Volcengine TOS bucket where the data to be migrated resides.
NoteThe bucket name cannot start or end with invalid characters such as spaces, line breaks, or tabs.
Prefix
No
You can specify a data path prefix to migrate part of the data.
Specify a prefix: migrates the data in the specified directory (prefix). The prefix cannot start with a forward slash (/) and must end with a forward slash (/).
Do not specify a prefix: migrates all data in the bucket.
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
UCloud
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
On the Create Data Address page, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select UCloud US3.
Domain Name
Yes
The endpoint domain name of the bucket. Custom domain names and CDNs are not supported. For more information, see Endpoint list.
You can log on to the UCloud console, click the name of the bucket that you want to migrate, and choose to view the region where the bucket resides.
ImportantThe preceding steps are for reference only and may be outdated due to changes on the source website.
Public Key
Yes
Enter the public key of the API key that is used for the migration. The key is deleted after the migration is complete.
Private Key
Yes
Enter the private key of the API key that is used for the migration. The key is deleted after the migration is complete.
Bucket
Yes
Select the bucket where the data to be migrated resides.
NoteThe bucket name cannot start or end with invalid characters such as spaces, line breaks, or tabs.
Prefix
No
You can specify a data path prefix to migrate part of the data.
Specify a prefix: migrates the data in the specified directory (prefix). The prefix cannot start with a forward slash (/) and must end with a forward slash (/).
Do not specify a prefix: migrates all data in the bucket.
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
Baidu BOS
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
On the Create Data Address page, configure the following parameters, and click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select Baidu BOS.
Domain Name
Yes
Enter the region where the source address resides. For more information, see Regions and Endpoints.
AccessKeyId
Yes
Enter the access key that is used for the migration. The key is deleted after the migration is complete.
SecretAccessKey
Yes
Bucket
Yes
The bucket name of Baidu Cloud BOS object storage.
The bucket name cannot start or end with invalid characters such as spaces, line breaks, or tabs.
Prefix
No
You can specify a data path prefix to migrate part of the data.
Specify a prefix: migrates the data in the specified directory (prefix). The prefix cannot start with a forward slash (/) and must end with a forward slash (/).
Do not specify a prefix: migrates all data in the bucket.
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
Kingsoft KS3
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
On the Create Data Address page, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select Kingsoft KS3.
Domain Name
Yes
Enter the domain name that corresponds to the region where the bucket resides. Custom domain names and CDNs are not supported. For more information, see Mapping between endpoints and regions.
AccessKeyID
Yes
Enter the access key that is used for the migration. We recommend that you create a key for the migration service and delete the key after the migration is complete. For more information, see Documentation.
SecretAccessKey
Yes
Bucket
Yes
For a Kingsoft Cloud object storage bucket, enter only the custom name. For example, if the bucket name is
exampletest-1234567890, enterexampletest.Prefix
No
You can specify a data path prefix to migrate part of the data.
Specify a prefix: migrates the data in the specified directory (prefix). The prefix cannot start with a forward slash (/) and must end with a forward slash (/).
Do not specify a prefix: migrates all data in the bucket.
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
Google Cloud Storage
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
On the Create Data Address page, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select Google Cloud Storage.
Request endpoint (Cloud Storage URI)
Yes
The request endpoint (Cloud Storage URI) of the bucket. Custom domain names and CDNs are not supported. You can query the request endpoint on the Interoperability page of the Settings panel in the Cloud Storage console.
AccessKey ID
Yes
To read the data to be migrated, you must provide the AccessKey pair of the source data account. You can query the AccessKey pair on the Interoperability page of the Settings panel in the Cloud Storage console.
NoteSecurity tip: To ensure data security, we strongly recommend that you create a separate key with [Read-only] permissions for the migration service and delete the key after the migration task is complete.
AccessKey Secret
Yes
To read the data to be migrated, you must provide the access key of the source data. You can query the secret of the user account on the Interoperability page of the Settings panel in the Cloud Storage console.
NoteSecurity tip: To ensure data security, we strongly recommend that you create a separate key with [Read-only] permissions for the migration service and delete the key after the migration task is complete.
Bucket
Yes
Enter the name of the Google Cloud Storage bucket where the data to be migrated resides.
NoteThe bucket name cannot start or end with invalid characters such as spaces, line breaks, or tabs.
Prefix
No
You can specify a data path prefix to migrate part of the data.
Specify a prefix: migrates the data in the specified directory (prefix). The prefix cannot start with a forward slash (/) and must end with a forward slash (/).
Do not specify a prefix: migrates all data in the bucket.
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
Youpai USS
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
On the Create Data Address page, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select Youpai USS.
Domain Name
Yes
The endpoint domain name of the bucket. For more information, see Service domain names.
AccessKey
Yes
The AWS S3 access credential. Authentication is performed based on the S3 protocol standard to verify whether the credential has the permissions to read data from the bucket. The
AccessKeyandSecretAccessKeythat are required to calculate the signature can be obtained in the console from "Operator - Edit" or "Cloud Storage - Configuration - Storage Management - Operator Authorization - S3 Access Credential". For more information, see AWS S3 Compatibility.SecretAccessKey
Yes
Prefix
Yes
Specify a prefix to migrate a subset of the data.
-
If specified, only data under that prefix is migrated. The prefix must not start with a forward slash (/) but must end with one.
-
If you omit this parameter, all data in the bucket is migrated.
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
Azure Blob
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
On the Create Data Address page, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select Azure Blob.
Domain Name
No
The endpoint of Azure Blob. Custom domain names and CDN are not supported. Example: Storage Account.blob.core.windows.net.
SAS Token
Yes
To read the data to be migrated, you must provide a SAS token. View the SAS token on the shared access signature page of the storage account in the console.
NoteSecurity tip: To ensure data security, we strongly recommend that you create a separate SAS token with [Read-only] permissions for the migration service and revoke the token after the migration task is complete.
Container
Yes
Enter the name of the Azure Blob container where the data to be migrated resides.
NoteThe container name cannot start or end with invalid characters such as spaces, line breaks, or tabs.
Prefix
No
You can specify a data path prefix to migrate part of the data.
Specify a prefix: migrates the data in the specified directory (prefix). The prefix cannot start with a forward slash (/) and must end with a forward slash (/).
Do not specify a prefix: migrates all data in the bucket.
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
LocalFS
If the LocalFS source data is stored in local files on the machine, you can deploy only one agent.
If the LocalFS source data is stored in local files mounted from Apsara File Storage NAS and is mounted on multiple machines, make sure that the NAS file system is mounted to directories with the same name. When you create the data address, set the directory to be migrated to the directory to which the NAS file system is mounted.
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
In the Create Address panel, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select LocalFS.
Directory To Be Migrated
Yes
You can specify the path of the directory to be migrated.This path must begin and end with a forward slash (/) and cannot contain environment variables or special characters.
For example, if you set the prefix of the source data address to
/example/src/and this path contains the file example.jpg. If the prefix of the destination data address is set toexample/dest/, the full path of the file example.jpg after migration isexample/dest/example.jpg.ImportantIf the data address is associated with multiple agents, make sure that each agent can access the directory. Otherwise, some data is not migrated.
If the LocalFS source data is stored in local files mounted from Apsara File Storage NAS and is mounted on multiple machines, make sure that the NAS file system is mounted to local directories with the same name. When you create the data address, set the directory to be migrated to the name of the local directory to which the NAS file system is mounted.
Tunnel
Yes
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
Yes
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
OSSHDFS
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
In the Create Address panel, configure the following parameters, and then click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select OSSHDFS.
Domain Name
Yes
Enter the endpoint that is used to access OSS-HDFS.
Format: <region-id>.oss-dls.aliyuncs.com. For example, the endpoint of the China (Hangzhou) region is cn-hangzhou.oss-dls.aliyuncs.com.
AccessKeyId
Yes
Enter the access key of the IAM user that you created. The access key is used to identify OSS-HDFS. Make sure that the user has write permissions on the destination data.
SecretAccessKey
Yes
Bucket
Yes
Enter the name of the bucket that stores the data to be migrated in the current account.
Directory To Be Migrated
Yes
To migrate part of the data, specify a data path that starts and ends with a forward slash (/), such as /data/to/osshdfs/. To migrate all files in the bucket, set the value to /.
Tunnel
No
Select the name of the tunnel that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from FTP, HDFS, or self-managed storage to the cloud.
You must associate an agent when the destination data address is of the LOCALFS type, or when the migration must go through an Express Connect circuit, such as in Finance Cloud or Apsara Stack scenarios.
Agent
No
Select the name of the agent that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from FTP, HDFS, or self-managed storage to the cloud.
You can select up to 30 agents at a time for a specified tunnel.
FTP
In the left-side navigation pane, choose Data Online Migration > Address Management, and then click Create Address.
In the Create Data Address panel, configure the following parameters, and click OK.
Parameter
Required
Description
Name
Yes
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select FTP.
Connection Address
Yes
The IP address and port number of the FTP server. The address of the FTP server is a concatenation of the IP address and the port number. That is, the value of this parameter is in the {hostname or IP address}:{port} format, for example, 192.168.16.13:21.
Directory To Be Migrated
Yes
You can specify the path of the directory to be migrated.This path must begin and end with a forward slash (/) and cannot contain environment variables or special characters.
For example, if you set the prefix of the source data address to
/example/src/and this path contains the file example.jpg. If the prefix of the destination data address is set toexample/dest/, the full path of the file example.jpg after migration isexample/dest/example.jpg.
Username
Yes
Enter the FTP username and password.
Password
Yes
Tunnel
Yes
Select the name of the tunnel that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from FTP or self-managed storage to the cloud.
You must associate an agent when the destination data address is of the LOCALFS type, or when the migration must go through an Express Connect circuit, such as in Finance Cloud or Apsara Stack scenarios.
Agent
Yes
Select the name of the agent that you want to use.
ImportantThis parameter is required only when you migrate data to the cloud over an Express Connect circuit or a VPN, or when you migrate data from FTP or self-managed storage to the cloud.
You can select up to 30 agents at a time for a specified tunnel.
Step 3: Create a destination address
In the navigation pane on the left, choose Data Online Migration > Address Management, and then click Create Address.
In the Create Address panel, configure the following parameters, and then click OK.
The name must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name cannot start with a hyphen (-) or an underscore (_).
The destination bucket belongs to the account that is used to log on to the Data Online Migration console
The destination bucket does not belong to the account that is used to log on to the Data Online Migration console
Specify a prefix: for example, if the prefix of the source data address is
example/src/and this path contains the file example.jpg. If the prefix of the destination data address is set toexample/dest/, the full path of the file example.jpg after migration isexample/dest/example.jpg.Do not specify a prefix: if you do not set a data path prefix, the source data is migrated to the root directory of the destination bucket.
This parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
This parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
Parameter | Required | Description |
Name | Yes | Enter a name for the destination data address. The naming rules are as follows: |
Type | Yes | Select Alibaba OSS. |
Custom Domain Name | No | Custom domain names are supported. |
Region | Yes | Select the region where the destination address resides, for example, China (Hangzhou). |
Authorize Role | Yes | |
Bucket | Yes | Enter the name of the bucket that stores the migrated data in the current account. |
Prefix | No | You can specify a data path prefix to migrate the source data to a specified directory. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, |
Tunnel | No | Select the name of the channel to use. Important |
Agent | No | Select the name of the agent to use. Important |
Step 4: Create a migration task
In the left-side navigation pane, choose Data Online Migration > Migration Tasks, and then click Create Task.
On the Select Address page, configure the following parameters, and then click Next.
Parameter
Required
Description
Name
Yes
Enter a name for the migration task. The naming rules are as follows:
The name cannot be empty and must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name is encoded in UTF-8 and cannot start with a hyphen (-) or an underscore (_).
Source Address
Yes
Select the source address that you created.
Destination Address
Yes
Select the destination address that you created.
On the Task Configurations page, configure the following parameters.
Parameter | Required | Description |
Basic Settings | ||
Migration Bandwidth | No | Select the migration bandwidth.
Important
|
Files Migrated Per Second | No | Select the number of files migrated per second.
Important
|
Overwrite Mode | Yes | Select the overwrite mode for files with the same name.
Warning |
Audit Mode | ||
Migration Report | Yes | The method that is used to push migration reports.
Important
|
Migration Logs | Yes | The method that is used to push migration logs.
When you select Push or Push only file error logs., Data Online Migration creates a project in Log Service named aliyun-oss-import-log-{Alibaba Cloud account ID}-{current region where the migration service is deployed}. Example: aliyun-oss-import-log-137918634953****-cn-hangzhou. Important Make sure that you have completed the following operations before you select Push or Push only file error logs.. Otherwise, the migration task may run abnormally.
|
Authorize | No | This option is displayed when Migration Logs is set to Push or Push only file error logs.. Click Authorize to go to the Cloud Resource Access Authorization page. The system creates the AliyunOSSImportSlsAuditRole role on the page and grants the required permissions to the role. Click Agree to Authorize to complete the authorization. |
Filter | ||
File Name | No | The file name filter. Two filter rules are supported: Include and Exclude. For more information about the regular expression syntax, see the RE2 library. Only part of the syntax is supported. For example:
Important
|
File Modification Time | No | The filter based on the last modified time of files. You can specify the last modified time of files as a filter rule. If you specify a time range, only files whose last modified time falls within the specified range are migrated. The rules are as follows:
|
Whether to Migrate | No | Important This option is supported only when the source is LocalFS. Specifies whether to migrate entities of special types. If you select the check box, the migration is enabled. If you clear the check box, the migration is disabled. Directory:
Symbolic Link:
Important In all cases, the target files or directories to which the symbolic link files at the source address point are not additionally migrated, unless the target files or directories are also within the scope of this migration. |
Migration Configurations | ||
Convert Destination Path Or Not | No | Important This option is supported only when the source is LocalFS. Converts the Target attribute of the symbolic link files at the source address so that the migrated Symlink objects can still point to their target objects wherever possible. If you select the check box, the conversion is enabled. If you clear the check box, the conversion is disabled. Important
When this option is enabled, the Target attribute is first resolved into the shortest absolute path, that is, the equivalent path with the fewest characters (AbsTarget), relative to the directory where the symbolic link resides. A string replacement is then performed to replace SrcPrefix in AbsTarget with DestPrefix if SrcPrefix is matched. The replaced value is set as the Target attribute of the destination Symlink object. Description Example: assume that SrcPrefix is set to "/mnt/nas1/" and DestPrefix is set to "cloud_base/" for the migration task, and that /mnt/nas1/links/a.lnk is a symbolic link file at the source address. Assume that the Target attribute of the symbolic link file:
When this option is disabled, no conversion is performed, and the original value of the Target attribute of the source symbolic link is set as the Target attribute of the destination Symlink object. |
Retain Last Modified Time | Yes | Specifies whether to retain the last modified time of source files.
|
Convert from Appendable to Normal or Multipart | No | Specifies whether to convert files of the Appendable type to the Normal or Multipart type.
Important
|
Add Target Storage Class | No | Specifies whether to specify the storage class of the destination files.
Important
|
Task Scheduling | ||
Execution Time | No | Important
Specifies the execution time of the migration task.
Important You can manually start and pause the task at any time, regardless of the custom execution time. |
Read the Online Migration Service Agreement, select I have read and agree to the Compliance Commitment and that when the migration task is completed, I have the obligation and responsibility to confirm the consistency of the migrated data, and then click Next.
Confirm that the configuration is correct, and then click OK. Wait for the migration task to run.
Execution frequency
Execution frequency | Description | Example |
Hourly | Run the task once every hour. You can use this option with the maximum number of runs. | The current time is 8:05. The frequency is set to hourly with a maximum of 3 runs. The first run starts at the next hour, 9:00.
|
Daily | Run the task once a day. You must specify an hour (0-23) for the task to start. You can use this option with the maximum number of runs. | The current time is 8:05. The task is scheduled to run daily at 10:00, with a maximum of 5 runs. The first run starts at 10:00 today.
|
Weekly | Run the task once a week. You must specify a day of the week and an hour (0-23) for the task to start. You can use this option with the maximum number of runs. | The current time is Monday, 8:05. The task is scheduled to run every Monday at 10:00, with a maximum of 10 runs. The first run starts at 10:00 today.
|
Specific days of the week | Run the task on selected days of the week. You must specify the days and an hour (0-23) for the task to start. | The current time is Wednesday, 8:05. The task is scheduled to run on Mondays, Wednesdays, and Fridays at 10:00. The first run starts at 10:00 today.
|
Custom | Use a cron expression to define a custom schedule for the task start time. | Note A cron expression consists of six space-separated fields that define the execution schedule: second, minute, hour, day of the month, month, and day of the week. The minimum interval is 1 hour. The following cron expression examples are for reference only. For more options, use a cron expression generator.
|
Step 5: Verify data
The migration service only transfers your data and does not guarantee data consistency or integrity. After the migration task completes, you must verify data consistency between the source and the destination.
After the migration task completes, you must verify the data at the destination. Verify that all of your data has been completely migrated to the destination before deleting the source data. If you delete the source data without verifying it, you are responsible for any resulting data loss.