Preparations
Before you use Data Online Migration to migrate data to Alibaba Cloud Object Storage Service (OSS), prepare the account, data source, and destination data address in sequence.
Step 1: Prepare an account
1. Create a RAM user
Use a Resource Access Management (RAM) user to operate the Data Online Migration service. (Recommended) Create the RAM user in the Alibaba Cloud account to which the source or destination OSS bucket belongs. The account that you choose becomes the current account and determines whether each bucket in Steps 2 and 3 uses the same-account or cross-account authorization branch. Use this RAM user to log on when you subsequently create RAM roles, data addresses, and migration jobs.
If you have not created a RAM user, see Create and authorize a RAM user. If you already have a RAM user, configure the following permissions for the user.
2. Grant permissions to the RAM user
Grant the following permissions to the RAM user:
Permission | Purpose |
The | Manage Data Online Migration. |
A custom policy that contains the following actions | Create and configure the RAM roles and access policies used for migration. |
The following code shows the custom policy. For information about how to create the policy, see Create a custom policy.
{
"Version": "1",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ram:CreateRole",
"ram:CreatePolicy",
"ram:AttachPolicyToRole",
"ram:ListRoles"
],
"Resource": "*"
}
]
}Step 1 configures permissions for the account that operates the Data Online Migration service. The permissions that Data Online Migration requires to read source data, read inventory files, and write data to the destination OSS bucket are prepared separately in Steps 2 and 3.
Step 2: Prepare the data source
Select the tab for your data source type and complete the preparations:
If you migrate data based on an inventory, use the Inventory tab. This tab applies to the Inventory, OSS Inventory, S3 Inventory, and HTTP/HTTPS source types. Configure the access permissions for the source data as described in the tab for the storage service in which the source data is stored.
Otherwise, use the tab for the storage service that holds the source data: Third-party object storage, Alibaba Cloud OSS, LocalFS, or OSS-HDFS.
Third-party object storage
This section applies to third-party object storage services, such as Amazon S3, Tencent Cloud Object Storage (COS), and Huawei Cloud Object Storage Service (OBS). For an inventory-based migration, see the Inventory tab in this step.
Item | Description |
Storage location | Prepare the source bucket name and its region, endpoint, or other access address. |
Access credentials | Prepare the AccessKey, SecretKey, SAS token, or other credentials required by the source type. |
Access permissions | Make sure that the credentials have permissions to list the objects to migrate and read object content and required metadata. |
Migration scope | Determine whether to migrate the entire bucket or only the objects with a specified prefix. |
Parameter names and credential formats vary by provider. For the requirements of each source type, see the corresponding section in Migration implementation.
Alibaba Cloud OSS
This section applies to OSS sources and to scenarios in which data in OSS is migrated by using OSS Inventory or Inventory.
1. Prepare the source bucket information
Record the source bucket name, region, and prefix of the objects to migrate, and determine whether the bucket belongs to the current account.
2. Configure permissions on the source bucket
Select an authorization method based on whether the source bucket belongs to the current account. The current account is the Alibaba Cloud account to which the RAM user created in Step 1 belongs.
The source bucket belongs to the current account
Automatic authorization (recommended)
When you create the source data address, complete automatic authorization in Authorization Role. You do not need to manually create a role in advance.
Manual authorization
Use manual authorization if you need to reuse a role to manage multiple buckets, the number of roles is limited, or automatic authorization is unavailable.
Create a RAM role
Use the RAM user prepared in Step 1 to log on to the RAM console and create a role on the Roles page:
Set Principal Type to Cloud Service.
Set Principal Name to Data Transport (
mgw.aliyuncs.com), the service principal of Data Online Migration.Enter a role name, such as
role-test, and create the role.A role name can contain lowercase letters, digits, hyphens (
-), and periods (.), and can be up to 64 characters in length. If an existing role meets the requirements, you can grant the required permissions to that role.
Grant read permissions on the source bucket
Create the following custom policy and attach it to the RAM role. Replace <srcBucket> with the actual source bucket name.
{
"Version": "1",
"Statement": [
{
"Effect": "Allow",
"Action": ["oss:List*", "oss:Get*"],
"Resource": [
"acs:oss:*:*:<srcBucket>",
"acs:oss:*:*:<srcBucket>/*"
]
}
]
}The source bucket does not belong to the current account
Create a RAM role
Use the RAM user prepared in Step 1 to create a RAM role in the current account as described in Create a RAM role under Manual authorization in The source bucket belongs to the current account on the Alibaba Cloud OSS tab of this step.
Configure a bucket policy for the source bucket
Use the account to which the source bucket belongs to log on to the OSS console. Select the source bucket and choose Permissions > Bucket Policy. On the Add by Syntax tab, add the following policy.
Placeholder | Value |
| The UID of the Alibaba Cloud account to which the RAM user in Step 1 belongs. |
| The UID of the Alibaba Cloud account to which the source bucket belongs. |
| The name of the RAM role created in the current account. |
| The source bucket name. |
{
"Version": "1",
"Statement": [
{
"Effect": "Allow",
"Action": ["oss:List*", "oss:Get*"],
"Principal": [
"arn:sts::<migrationUid>:assumed-role/<roleName>/*"
],
"Resource": [
"acs:oss:*:<bucketOwnerUid>:<srcBucket>",
"acs:oss:*:<bucketOwnerUid>:<srcBucket>/*"
]
}
]
}When you edit the bucket policy, retain the existing policies and merge the new authorization into them to prevent impacts on existing access permissions. The Principal uses the STS assumed-role ARN format (arn:sts::<migrationUid>:assumed-role/<roleName>/*), which is not interchangeable with the RAM role ARN format (acs:ram::<migrationUid>:role/<roleName>) used in a KMS key policy.
3. Configure KMS permissions if required
If the source objects to migrate are encrypted by using Key Management Service (KMS), attach the AliyunKMSFullAccess system policy to the RAM role that reads the source data.
For cross-account access to objects encrypted by using a customer-managed KMS key, the account that owns the key must also configure the key policy and add the ARN of the role to Other Account Users:
acs:ram::<migrationUid>:role/<roleName>The key policy uses the RAM role ARN format, which is not interchangeable with the STS assumed-role ARN format used in the bucket policy. For more information, see Configure a key policy.
Inventory
This section applies to Inventory, OSS Inventory, S3 Inventory, and HTTP/HTTPS sources. Prepare the following items:
Item | Description |
Source data access | Prepare the access credentials and read permissions for the objects listed in the inventory. For an HTTP/HTTPS source, provide accessible file URLs. |
Inventory files | Use a native inventory generated by the storage provider or create a custom inventory in the format required by Data Online Migration. |
Inventory access | Record the inventory location and configure permissions to read the inventory files. |
The source bucket stores the data to migrate, and the inventory bucket stores the inventory files. If they are different buckets, configure access permissions for both. If they are the same bucket, grant permissions only once.
1. Prepare permissions to access the source data
If the source data listed in the inventory is stored in a third-party object storage service, prepare the connection information, access credentials, and read permissions for that service.
If the source data listed in the inventory is stored in OSS, complete authorization as described in Configure permissions on the source bucket on the Alibaba Cloud OSS tab of this step.
For an HTTP/HTTPS source, each file URL must support HTTP HEAD requests for metadata retrieval and HTTP GET requests for content download.
2. Prepare the inventory files
The inventory files that you prepare depend on the inventory type that you select when you create the source data address.
Inventory
Generic Inventory supports custom inventories and the native inventories of supported storage providers, including OSS and Amazon S3.
Supported sources and native inventories
Generic Inventory supports the following 13 object storage sources. The following table describes native inventory support.
Object storage source | Native inventory supported |
Alibaba Cloud OSS | Yes |
Amazon S3 | Yes |
Microsoft Azure Blob Storage | Yes |
Google Cloud Storage | Yes |
Qiniu Kodo | Yes |
S3-compatible third-party storage (Compatible S3) | No. Use a custom inventory. |
Tencent Cloud COS | Yes |
Huawei Cloud OBS | Yes |
Baidu Object Storage (BOS) | Yes |
Volcano Engine TOS | Yes |
Kingsoft Cloud KS3 | Yes |
UPYUN USS | No. Use a custom inventory. |
UCloud US3 | No. Use a custom inventory. |
To use a native inventory, configure an inventory generation rule in the source storage service and wait for the inventory files to be generated. To use a custom inventory, create the manifest.json and .csv.gz list files as described in Custom inventory format under Inventory list format reference later in this tab.
HTTP/HTTPS
Verify that file URLs are accessible
Data Online Migration downloads files by using HTTP GET requests and retrieves file metadata by using HTTP HEAD requests. Each original URL in the list must directly support both request types. Use URLs that are accessible without redirection.
To create the manifest.json and .csv.gz list files, see HTTP/HTTPS inventory format under Inventory list format reference later in this tab.
OSS Inventory
Configure a bucket inventory for the source OSS bucket and wait for the inventory files to be generated. For more information, see Bucket inventory.
Record the bucket and path in which the inventory is stored, and prepare the permissions required to read the source objects and inventory files. When you create the source data address, enter the inventory parameters for the OSS Inventory source type.
S3 Inventory
Configure an inventory for the source Amazon S3 bucket and wait for the inventory files to be generated. For more information, see Configuring Amazon S3 Inventory.
Record the bucket, region, and path in which the inventory is stored, and prepare the access credentials and permissions required to read the source objects and inventory files. When you create the source data address, enter the inventory parameters for the S3 Inventory source type.
3. Configure access to the inventory
Record the storage service, bucket, region or access domain, and the path of manifest.json. Make sure that Data Online Migration can read the inventory and all list files referenced by the inventory.
For an Inventory or HTTP/HTTPS source, configure authorization based on the inventory location:
The inventory is stored in OSS in the current account
When you create the source data address, use List Authorization Role to complete automatic authorization.
For manual authorization, create or select a RAM role in the current account. Use the read policy in Grant read permissions on the source bucket under Manual authorization in The source bucket belongs to the current account on the Alibaba Cloud OSS tab of this step, and replace <srcBucket> with the inventory bucket name to grant list and read permissions.
The inventory is stored in OSS in another account
Use the account that owns the inventory bucket to configure a bucket policy that allows a RAM role in the current account to read the inventory.
Use the bucket policy in Configure a bucket policy for the source bucket in The source bucket does not belong to the current account on the Alibaba Cloud OSS tab of this step. Replace <srcBucket> with the inventory bucket name and <bucketOwnerUid> with the UID of the Alibaba Cloud account that owns the bucket.
The inventory is stored in another storage service
Prepare the access credentials for the corresponding storage service and grant permissions to read manifest.json and all list files referenced by it. For supported storage types and parameter requirements, see the corresponding source type in Migration implementation.
An HTTP/HTTPS inventory can be stored in OSS or Amazon S3. If the inventory is stored in OSS, configure authorization as described in the OSS tabs of this step. If the inventory is stored in Amazon S3, prepare the Amazon S3 access credentials and read permissions as described in this tab. For an Inventory source, store the list in an inventory storage type supported by the console.
If inventory files in OSS are encrypted by using KMS, also configure permissions for the RAM role that reads the inventory and for the corresponding key, as described in Configure KMS permissions if required on the Alibaba Cloud OSS tab of this step.
For the inventory access parameters of OSS Inventory and S3 Inventory, see the requirements of the corresponding source type in Migration implementation.
Inventory list format reference
Use the following formats when you create inventory files for an Inventory or HTTP/HTTPS source.
Custom inventory format
The following format applies to custom inventories used with an Inventory source. For a native inventory, use the inventory feature provided by the storage provider.
A custom inventory consists of one manifest.json file and one or more .csv.gz list files. Each list file records a batch of objects to migrate, and manifest.json describes the locations and field formats of the list files.
Create CSV list files
Each CSV row records one object. The columns are separated by commas and the fields are defined as follows:
Field | Required | Description |
| Yes | The full name of the object in the source bucket. Enter the full name and URL-encode it, regardless of whether a prefix is configured for the source data address. |
| No | The source object size in bytes. This field is used to calculate the migrated data volume. If this field is omitted, the storage volume chart is unavailable in the console. |
| No | The destination object name after migration, URL-encoded. If a destination prefix is configured, the final name is the destination prefix followed by the decoded |
URL encoding rules
URL-encode the actual source object name once and write it to Key. URL-encode the intended destination object name once and write it to DestKey.
Actual object name | Value in the CSV file |
|
|
|
|
|
|
A plus sign, a percent sign, and a comma are encoded as %2B, %25, and %2C, respectively. If an object name contains percent-encoded text, preserve the actual meaning of the name and then encode the entire name.
File format requirements
Use UTF-8 encoding without a byte order mark (BOM).
End every row, including the last row, with a line feed.
Make sure that every row has the same number of columns and that the number and order match the fields in
fileSchemainmanifest.json.Include only object records in the CSV file. Do not add a header row. Field names are declared in
fileSchema.For example, a two-column CSV file that uses
Key,Sizecontains the following data:
path%2Fto%2F1.txt,1024
a%2Bb%25c%2Cd.file,2048To specify destination object names, use three columns in the order Key,Size,DestKey:
path%2Fto%2F1.txt,1024,archive%2F1.txtIn this case, set fileSchema to Key,Size,DestKey. If the destination prefix is backup/, the object name after migration is backup/archive/1.txt.
Compress the CSV files
Compress each CSV file into a separate .csv.gz file. A compressed file cannot exceed 100 MB.
gzip -c example.csv > example.csv.gzThis command retains the original example.csv file. If multiple CSV files exist, compress them separately and list all compressed files in the files array of manifest.json.
Create manifest.json
The following sample corresponds to a CSV file that contains the Key,Size columns:
{
"fileFormat": "CSV",
"fileSchema": "Key,Size",
"files": [
{
"key": "inventory/example.csv.gz",
"MD5checksum": "",
"size": 0
}
],
"mgwInventoryVersion": "1.0"
}Field | Description |
| The list format. Set the value to |
| The names of the CSV fields, in the same order as the columns. |
| The array of list files. You can specify multiple files. |
| The object name of the |
| The hexadecimal MD5 hash of the list file. The value is case-insensitive. If this field is empty, the check is skipped. |
| The list file size. If this field is |
| The custom inventory format version. Set the value to |
A custom inventory for an Inventory source must contain mgwInventoryVersion. If this field is omitted, the service parses the files as a native inventory generated by a storage provider, which may cause parsing to fail.
Upload and record the inventory location
Upload manifest.json and all .csv.gz files to the inventory storage location. Make sure that each files[].key value matches the actual uploaded path, and record the path of manifest.json.
After the upload is complete, configure authorization as described in Configure access to the inventory in this tab.
Encoding example
The following Python 3 example reads the unencoded plain_example.csv file and generates an example.csv file in Key,Size format. The input file does not contain a header row. The script validates that each input row contains exactly two columns (Key and Size). To process a different column set, adjust the row validation in the script.
If an original name contains a comma, enclose the field in double quotation marks as required by CSV syntax. Example:
path/to/1.txt,1024
"a+b%c,d.file",2048import csv
from urllib.parse import quote_plus
with open("plain_example.csv", encoding="utf-8-sig", newline="") as src, \
open("example.csv", "w", encoding="utf-8", newline="") as dst:
reader = csv.reader(src)
writer = csv.writer(dst, lineterminator="\n")
for line_number, row in enumerate(reader, start=1):
if len(row) != 2 or not row[0]:
raise ValueError(f"Row {line_number} must contain non-empty Key and Size fields")
key, size = row
writer.writerow([quote_plus(key, safe=""), size])The example uses a CSV parser to process fields that contain quotation marks and commas. To add DestKey, add the corresponding column to the input file and apply the same URL encoding to that column. Update fileSchema accordingly.
HTTP/HTTPS inventory format
An HTTP/HTTPS inventory consists of one manifest.json file and one or more .csv.gz list files. Each row specifies the URL of a file and the destination object name after migration.
Create CSV list files
URL-encode each accessible full URL once and write it to the Url field in the CSV file. Percent signs in the original URL must also be encoded.
For example, if the path in an accessible URL contains %E6%96%87, the encoded result is %25E6%2596%2587.
Field | Required | Description |
| Yes | The complete file URL, URL-encoded once. |
| Yes | The intended destination object name, URL-encoded once. If a destination prefix is configured, the final name is the destination prefix followed by the decoded |
| No | The file size in bytes. This field is used to calculate the migrated data volume. If this field is omitted, the storage volume chart is unavailable in the console. |
In an Inventory list, Key specifies the source object name. In an HTTP/HTTPS list, Key specifies the destination object name. Enter the value based on the selected source type.
For example, if the original file URL is https://example.com/files/1.txt and the intended destination object name is archive/1.txt, a row in a two-column Url,Key CSV file is as follows:
https%3A%2F%2Fexample.com%2Ffiles%2F1.txt,archive%2F1.txtThis URL is provided only to demonstrate the format. Replace it with an actual accessible file URL.
Use UTF-8 encoding without a BOM for the CSV file, and do not include a header row. End each row with a line feed. Make sure that the number and order of columns in every row match fileSchema. For the encoding rules for plus signs, percent signs, and commas, see URL encoding rules in Custom inventory format.
Compress list files and create manifest.json
Compress each CSV file into a separate .csv.gz file as described in Compress the CSV files in Custom inventory format. A compressed file cannot exceed 100 MB.
The following manifest.json file corresponds to a CSV file that contains the Url,Key columns:
{
"fileFormat": "CSV",
"fileSchema": "Url,Key",
"files": [
{
"key": "inventory/example.csv.gz",
"MD5checksum": "",
"size": 0
}
]
}For descriptions of fileFormat, fileSchema, and files, see the field table in Create manifest.json in Custom inventory format. For an HTTP/HTTPS inventory, use the preceding sample and do not add mgwInventoryVersion, the version field that a custom inventory for an Inventory source requires.
Upload the inventory
Upload manifest.json and all referenced .csv.gz files to OSS or Amazon S3. Make sure that each list file path matches its files[].key value, and record the path of manifest.json.
After the upload is complete, configure the permissions required to read the inventory as described in Configure access to the inventory in this tab.
Encoding example
The following Python 3 example reads the unencoded plain_example.csv file and generates a two-column Url,Keyexample.csv file. The input file does not contain a header row. The script validates that each input row contains exactly two columns. Enclose fields that contain commas in double quotation marks as required by CSV syntax.
import csv
from urllib.parse import quote_plus
with open("plain_example.csv", encoding="utf-8-sig", newline="") as src, \
open("example.csv", "w", encoding="utf-8", newline="") as dst:
reader = csv.reader(src)
writer = csv.writer(dst, lineterminator="\n")
for line_number, row in enumerate(reader, start=1):
if len(row) != 2 or not all(row):
raise ValueError(f"Row {line_number} must contain non-empty Url and Key fields")
url, key = row
writer.writerow([quote_plus(url, safe=""), quote_plus(key, safe="")])LocalFS
LocalFS includes local disks and file systems such as NAS and Cloud Parallel File Storage (CPFS) that are mounted locally. Determine the source directory to migrate and complete the following preparations.
Item | Description |
Source directory | Make sure that the directory exists and record its absolute path, such as |
File system mount | If the source data is stored in a file system such as NAS or CPFS, mount the file system first and make sure that the files in the mount directory are accessible. |
Read permissions | Make sure that the operating system user that runs the migration agent can traverse the source directory and its subdirectories and read the files to migrate. |
Multi-agent access | If multiple agents are associated with the same source data address, make sure that all agents use the specified directory path to access the same source data. |
RAM user permissions are used to operate the Data Online Migration console. Access to local directories and files is controlled by the operating system and file system, and must be configured separately.
OSS-HDFS
This section applies to scenarios in which data is migrated from OSS-HDFS to Alibaba Cloud OSS.
Before you create an OSS-HDFS data address, submit a ticket to apply for inclusion in the whitelist. For more information, see Background information.
Prepare the source bucket and directory information.
Item
Description
Source bucket
Record the name and region of the bucket for which OSS-HDFS is enabled.
HDFS endpoint
Obtain the HDFS endpoint from the Overview page of the source bucket, such as
cn-hangzhou.oss-dls.aliyuncs.com, and enter it in Domain when you create the source data address.Directory to migrate
Record the path of the directory to migrate, such as
/data/source/. The path must start and end with a forward slash (/). To migrate the entire bucket, enter/.Prepare access credentials and permissions. Use the account that owns the source bucket to create a RAM user and an AccessKey pair for the user. Record the AccessKey ID and AccessKey secret to enter when you create the source data address.
Grant the RAM user permissions to access OSS-HDFS. Make sure that the user can list the source directory and read the files to migrate. For more information, see Authorize a RAM user from a non-EMR cluster to access OSS-HDFS in Authorize access to OSS-HDFS.
The RAM user that operates Data Online Migration and the RAM user that accesses OSS-HDFS can be different users. When you create the OSS-HDFS data address, use the AccessKey pair of the authorized RAM user.
Step 3: Prepare the destination data address
1. Prepare the destination bucket
Select the OSS bucket that receives the migrated data. To create a bucket, see Create a bucket.
Record the destination bucket name, region, and owning account, and determine whether to write data to the root directory of the bucket or to a specified prefix. You must provide this information when you create the destination data address.
2. Configure access permissions on the destination bucket
Select an authorization method based on whether the destination bucket belongs to the current account. The current account is the Alibaba Cloud account to which the RAM user created in Step 1 belongs.
The destination bucket belongs to the current account
Automatic authorization (recommended)
When you create the destination data address, complete automatic authorization in Authorization Role. You do not need to manually create a role in advance.
Manual authorization
Use the RAM user prepared in Step 1 to create a RAM role in the current account as described in Create a RAM role under Manual authorization in The source bucket belongs to the current account on the Alibaba Cloud OSS tab in Step 2.
Create the following custom policy and attach it to the RAM role. Replace <destBucket> with the actual destination bucket name.
{
"Version": "1",
"Statement": [
{
"Effect": "Allow",
"Action": [
"oss:List*",
"oss:Get*",
"oss:Put*",
"oss:AbortMultipartUpload"
],
"Resource": [
"acs:oss:*:*:<destBucket>",
"acs:oss:*:*:<destBucket>/*"
]
}
]
}The destination bucket does not belong to the current account
Create a RAM role
Use the RAM user prepared in Step 1 to create a RAM role in the current account as described in Create a RAM role under Manual authorization in The source bucket belongs to the current account on the Alibaba Cloud OSS tab in Step 2.
Configure a bucket policy for the destination bucket
Use the account to which the destination bucket belongs to log on to the OSS console. Select the destination bucket and choose Permissions > Bucket Policy. On the Add by Syntax tab, add the following policy.
Placeholder | Value |
| The UID of the Alibaba Cloud account to which the RAM user in Step 1 belongs. |
| The UID of the Alibaba Cloud account to which the destination bucket belongs. |
| The name of the RAM role created in the current account. |
| The destination bucket name. |
{
"Version": "1",
"Statement": [
{
"Effect": "Allow",
"Action": [
"oss:List*",
"oss:Get*",
"oss:Put*",
"oss:AbortMultipartUpload"
],
"Principal": [
"arn:sts::<migrationUid>:assumed-role/<roleName>/*"
],
"Resource": [
"acs:oss:*:<bucketOwnerUid>:<destBucket>",
"acs:oss:*:<bucketOwnerUid>:<destBucket>/*"
]
}
]
}When you edit the bucket policy, retain the existing policies and merge the new authorization into them. The Principal uses the STS assumed-role ARN format (arn:sts::<migrationUid>:assumed-role/<roleName>/*), which is not interchangeable with the RAM role ARN format (acs:ram::<migrationUid>:role/<roleName>) used in a KMS key policy.
3. Configure KMS permissions if required
If server-side KMS encryption is enabled for the destination bucket, attach the AliyunKMSFullAccess system policy to the RAM role that accesses the destination bucket.
In a cross-account scenario, the account that owns the key must also add the migration role to the key policy and allow the role to use the key. The key policy uses the RAM role ARN format (acs:ram::<migrationUid>:role/<roleName>), which is not interchangeable with the STS assumed-role ARN format used in the bucket policy. For more information, see Configure a key policy.