Data address

Basic configurations

1. Name: A unique identifier for the data address. The name must follow these naming rules:

   • The name cannot be empty and must be 3 to 63 characters long.

   • The name can contain lowercase letters, numbers, hyphens (-), and underscores (_).

   • The name must be UTF-8 encoded and cannot start with a hyphen (-) or an underscore (_).

2. Type: The type of storage service for the data address. The following options are supported:

   • Object Storage Service supports mainstream cloud storage services such as Alibaba Cloud OSS, AWS S3, Tencent Cloud COS, Huawei Cloud OBS, Volcano Engine TOS, and Azure Blob.

   • File system:

     • Local File System (LocalFs): Supports local disk storage or mounted NAS or CPFS file systems.

   • Manifest:

     • HTTP manifest: A list of URLs that can be accessed over HTTP.

     • Object storage manifest: Such as Alibaba Cloud OSS manifests and Amazon S3 manifests.

Bucket configurations

1. Bucket: The name of the data storage container, such as an Amazon S3 bucket name.

2. AccessKey pair (Access Key ID / Secret Access Key): The key pair used for identity verification. For third-party cloud storage, the AccessKey pair must have read access to the source data.

3. Endpoint: The endpoint of the source storage service, such as the endpoint for Amazon S3.

4. Prefix: The path prefix for the files to be migrated or written. If you specify a prefix, the key of a file is changed. The following examples show how this works:

   • Prefix specified for the source data address: If the prefix for the source data address is set to example/src/ and no prefix is set for the destination data address, the full path of the file example/src/example.jpg at the destination becomes example.jpg after migration.

   • Prefix specified for the destination data address: If no prefix is set for the source data address and the prefix for the destination data address is set to example/dest/, the full path of the file example.jpg at the destination becomes example/dest/example.jpg after migration.

   • Prefixes specified for both the source and destination data addresses: If the prefix for the source data address is set to example/src/ and the prefix for the destination data address is set to example/dest/, the full path of the file example/src/example.jpg at the destination becomes example/dest/example.jpg after migration.

5. Region (RegionId): The ID of the region where the bucket is located, such as oss-cn-hangzhou. If the storage service is public cloud OSS, set this field and do not set an endpoint. Data Online Migration automatically selects a suitable endpoint.

6. Role: The RAM role used to access the data. If the storage service is public cloud OSS, set this field and do not set an AccessKey pair. Data Online Migration uses this role to read and write data, which reduces the security risks associated with using an AccessKey pair.

Manifest configuration

In a manifest-based migration scenario, you must configure the basic information for the manifest in addition to the bucket configurations.

1. Manifest Location (InvLocation): The storage location of the manifest file, such as OSS or Amazon S3.

2. Manifest Bucket (InvBucket): The bucket where the manifest file is located.

3. Manifest Endpoint (InvDomain): The endpoint for the manifest bucket.

4. Manifest Path (InvPath): The name of the manifest file. The manifest is a JSON file that defines the schema information and a series of CSV files.

5. Manifest AccessKey pair (InvAccessId / InvAccessSecret): The Access Key ID and Secret Access Key used to access the manifest bucket.

6. Manifest Region (InvRegionId): The ID of the region where the manifest bucket is located, such as oss-cn-hangzhou. If the manifest bucket is in public cloud OSS, set this field and do not set a manifest endpoint. Data Online Migration automatically selects a suitable endpoint.

7. Manifest Role (InvRole): The role used to access the manifest bucket. If the manifest bucket is in public cloud OSS, set this field and do not set an AccessKey pair. Data Online Migration uses this role to read the manifest, which reduces the security risks associated with using an AccessKey pair.