Copy local data-Data Transport(Data Transport)-阿里云帮助中心

Usage notes

Note the following when you migrate data using Data Transport:

When you specify the Directory To Be Migrated, use an absolute path. The path must start and end with a forward slash (/). Environment variables and special characters are not supported. Make sure that the Directory To Be Migrated exists and is valid.
A Data Transport migration consumes resources from the source, which can affect your regular business operations. If your services are critical, assess the impact in advance and either enable throttling on the source or run the data migration task during off-peak hours.
If a file with the same name exists in both the source path and the destination Object Storage Service (OSS) bucket, the migration defaults to Overwrite All. The source file overwrites the one in the destination bucket. If the two files have different content, you must rename one of them or back them up.

By default, Data Transport retains the last modified time attribute of a source file. Note that if a lifecycle rule is configured for the destination bucket and a migrated file falls within the effective time range of the lifecycle rule, the file may be deleted or converted to the specified storage class when the rule takes effect.
- If you do not need to preserve the last modified time, contact Alibaba Cloud support to change the setting. Be aware of the risks involved.

Limitations

Character device files, block device files, socket files, and pipe files from the source path are not migrated.
If your source path contains symbolic links, they are migrated only if the transfer_symlink parameter is set to true. Otherwise, they are skipped. For more information, see Symbolic links.
If a hard link exists in the source path, it is converted into a regular file after migration.
The attributes of parent directories are not migrated.
File permissions such as SUID, SGID, and SBID are not migrated.
The combined length of the directory path and filename cannot exceed 1024 bytes. The length of each directory name in the path cannot exceed 200 bytes, and the filename cannot exceed 255 bytes. Any files or directories that exceed these limits are not migrated.
Supported attributes include: Permissions -> X-Oss-Meta-Perms (OSS), Uid:Gid -> X-Oss-Meta-Owner (OSS), mtime -> X-Oss-Meta-Mtime (OSS), atime -> X-Oss-Meta-Atime (OSS), ctime -> X-Oss-Meta-Ctime (OSS).
Note
- Permissions: Includes nine permissions, such as read, write, and execute.
- Uid:Gid: The user ID and the group ID of the user, separated by a colon (:).
Unsupported attributes include, but are not limited to: AccessTime, Attr, and Acl.

Note
The migration behavior for unlisted attributes is not guaranteed.

Before you begin

Step 1: Decrypt and mount the storage pool

After you power on the device, the storage pool is locked. You can decrypt it with the initial secret, or you can add a new secret to use for decryption. To obtain the initial secret, contact Alibaba Cloud support.

To decrypt the storage pool with the initial secret, run the following command:
- Run crypt open <password>.
To add a new secret and decrypt the storage pool:
- Run crypt addsecret <old_password> <new_password>.
- To delete the initial secret, run crypt deletesecret <old_password>.
- To open the storage pool with the new secret, run crypt open <new_password>.

Note

We recommend that you create a new secret that contains uppercase and lowercase letters, digits, and special characters, and is at least eight characters long. For example: Hello@12345!.

Step 2: Initialize mgwserver

Important

Currently, only a single bucket is supported on a Data Transport III device. To create multiple buckets, contact Alibaba Cloud support.

Create a bucket, an AccessKey ID, and an AccessKey secret for the migration.

Verify that the mgwserver service is running properly. Run mgwserver status. If the status is abnormal, contact Alibaba Cloud support.
Create a bucket and set the AccessKey ID and AccessKey secret by running mgwserver create-bucket <Bucket> <AK> <SK>.
Note
- Bucket: The bucket name. We recommend that you use the same name as your OSS bucket. For example, xxx-xxx-data.
- AK: The AccessKey ID. This is analogous to a username. It can contain lowercase letters, digits, hyphens (-), and underscores (_). For example, aliyuncs@2024-xx-xx.
- SK: The AccessKey secret. This is analogous to a password. It can contain lowercase letters, digits, hyphens (-), and underscores (_). For example, xxxxxxxxx.

Create a migration task

Important

Each task must be specified at the directory level. If you need to migrate all data from the source, create only one data migration task. If you need to migrate data from N specific source directories, you must create N tasks. Confirm the number of tasks you need and create a corresponding number of copies of the demo-job.properties file. Each task requires its own properties file.

Configure the task file

Run the command cdmgwclient && cd conf to switch to the specified directory.
Confirm the number of tasks you need to create and create the required number of copies of the demo-job.properties file. Each task requires its own demo-job.properties file.
Note
- Make sure to differentiate the demo-job.properties files, for example: job1.properties and job2.properties.
- If you are migrating data from an entire directory, you only need to create one task.
- If you are migrating data from specific directories, create a separate task for each directory. The following steps use the job1.properties file as an example.

Open one of the properties files, such as job1.properties, and configure the following parameters:

Parameter	Required	Example value
job_name	Yes	The task name. For example: `example_job`.
src_path	Yes	The source path. The path must start and end with a forward slash (/). Example: The source path is `/mnt/nas/example_dir/`, which contains the file example.jpg. If the destination path is set to `mgw:example_bucket/dest/`, the full path of example.jpg after migration is `mgw:example_bucket/dest/example.jpg`.
dest_path	Yes	The destination path on the Data Transport device. The format is `mgw:BucketName/prefix/`. For example, `mgw:example_bucket/dest/`. BucketName: The name of the bucket used when initializing mgwserver. prefix: The prefix under the bucket. The prefix must end with a forward slash (/). Specify a prefix: For example, if the source path is `/mnt/nas/example_dir/`, which contains the file example.jpg, and the destination bucket prefix is set to `example/dest/`, the full path of example.jpg after migration is `mgw:example_bucket/example/dest/example.jpg`. Do not specify a prefix: For example, if the source path is `/mnt/nas/example_dir/`, which contains the file example.jpg, and no prefix is set for the destination bucket, the full path of the file after migration is `mgw:example_bucket/example.jpg`.
transfer_dir	Yes	Specifies whether to migrate directories. Set to `true` to enable migration, or `false` to disable it. If enabled, all directories scanned in the source path are added to the migration queue and included in the migration task's log file. An empty object whose name ends with / is created in the destination OSS bucket, and the migratable attributes of the source directory are stored in the UserMeta of the destination object. If disabled, all directories scanned in the source path are ignored and not included in the migration task's log file. No empty object whose name ends with / is created in the destination OSS bucket.
transfer_symlink	Yes	Specifies whether to migrate symbolic links. Set to `true` to enable migration, or `false` to disable it. If enabled, all symbolic link files in the source path are added to the migration queue and are counted in the file count and storage volume fields of the data migration task. A Symlink object is created in the destination OSS bucket, and the migratable attributes of the source symbolic link file are stored in the UserMeta of the Symlink object. For the Target attribute of the Symlink object, Data Transport first resolves the Target (relative to the directory that contains the symbolic link) into the shortest equivalent absolute path (AbsTarget). It then performs a string replacement: if `src_path` in AbsTarget matches, it is replaced with the `prefix` from `dest_path`. The result becomes the Target attribute of the destination OSS Symlink object. Note Example: Assume the migration task is configured with `src_path="/mnt/nas/example_dir/"`, the `prefix` in `dest_path` is `"example/dest/"`, and `/mnt/nas/example_dir/links/a.lnk` is a symbolic link file in the source path. If its Target attribute is: `"../data/./a.txt"`. The resolved shortest absolute path is `"/mnt/nas/example_dir/data/a.txt"`. The final Target value will be `"example/dest/data/a.txt"`. `"/mnt/nas/example_dir/verbose/../data/./a.txt"`. The resolved shortest absolute path is `"/mnt/nas/example_dir/data/a.txt"`, and the final Target value will be `"example/dest/data/a.txt"`. If disabled, all symbolic link files in the source path are ignored and are not counted in the file count and storage volume fields of the task. Important In all cases, the target file or directory pointed to by a symbolic link is not migrated unless that target is also within the scope of the migration.
transfers	No	The number of concurrent transfers for each task. The default is 16. For large file migrations, we recommend keeping the default value. For small file migrations, we recommend setting it to 256. The maximum value is 256.
log_level	No	The log level. The default is INFO. For more detailed logs, set it to DEBUG.

Start the task

Important

Note: Run cdmgwclient to enter the directory before you perform the following operations.

Run bash console.sh start conf/properties_filename to start the task. For example: bash console.sh start conf/job1.properties.

Check the task status

Important

Please note that all commands related to bash console.sh must be run from the cdmgwclient directory.

To check the task status:

Run bash console.sh status <job_name> to check the task status. Replace <job_name> with your actual task name.

If JobStatus is running, the task is in progress. The migration progress is displayed as follows:

JobStatus:        running
Transferred:      2.719 GiB / 2.878 GiB, 94%, 13.238 MiB/s, ETA 12s
Transferred:        2270 / 9307, 24%
ScanComplete:     true
 * xxx

If JobStatus is failed, some files failed to migrate. Investigate the cause of the failure based on the logs or contact Data Transport support for assistance.

JobStatus:        failed
Transferred:      29.270 KiB / 29.270 KiB, 100%, 0 B/s, ETA -
Transferred:          14 / 14, 100%
Checks:               10 / 10, 100%
Errors:                1 (retrying may help)
ScanComplete:     true
Elapsed time:     0.1s
You can view the error message in /home/aliyuncs/log/mgwclient/fxxx@xxxrt.log
You can view the transfers error message in /home/aliyuncs/log/mgwclient/xxxs-error.log

If JobStatus is succeed, all files have been migrated successfully. The total migrated data volume and file count are displayed as follows:
```
JobStatus:        succeed
Transferred:      126.335 GiB / 126.335 GiB, 100%, 4.462 MiB/s, ETA 0s
Transferred:        9303 / 9303, 100%
```

Note

The following describes the returned parameters:

First Transferred: Transferred data volume / Total data volume, migration percentage by volume, and average migration speed.
Second Transferred: Transferred file count / Total file count, and migration percentage by file count.
Checks: The number of skipped files. This is displayed only when files are skipped. If you need to remigrate some failed files from a previous task, files that were successfully migrated in the first task are counted as skipped files in the new task.
Errors: The number of failed files. This is displayed only when migration failures occur.
Elapsed time: The total migration duration.
ScanComplete: Indicates whether the scan is complete (true means complete, false means not complete).

Warning

If the source data is in a large, flat directory, the scan may take a long time, and the progress will update slowly.

Delete a task

To delete an incorrect task, run the following command to delete the specified task and its corresponding log directory:

Run bash console.sh delete <job_name> to delete the specified task. Replace <job_name> with the actual task name. Files that have already been migrated will not be deleted.

Next steps

Important

The following operations are performed in the /home/aliyuncs/log/mgwclient/job_name directory.

After the migration task is complete, if the status is failed, investigate the cause by checking the logs or contact Data Transport support.
If the migration task status is succeed, use the CRC-64 list to verify data consistency against the source files. This verification step is mandatory.

View logs

Important

In the following paths or filenames, replace job_name with the actual task name.

The /home/aliyuncs/log/mgwclient/job_name directory stores log files that record the upload status of each file and the overall status of the task.

Log name	Description
job_name-status.log	The task migration progress file.
job_name-folder.log	A list of migrated directories.
job_name-transfers-error.log	A list of files that failed to migrate.
job_name-crc64-succeed.log	A list of successfully migrated files with their CRC-64 values.
job_name-crc64-error.log	A list of files that failed CRC-64 verification during migration.
job_name-crc64-skipped.log	A list of files skipped during migration.

Important

Migration failures fall into two categories: transfer failures and CRC-64 verification failures. The total number of failed files is the sum of these two types.

Task progress log

The job_name-status.log file records basic information about the task's migration progress.
- First Transferred: Transferred data volume / Total data volume, migration percentage by volume, and average migration speed.
- Second Transferred: Transferred file count / Total file count, and migration percentage by file count.
- Checks: The number of skipped files. This is displayed only when files are skipped. If you need to remigrate some failed files from a previous task, files that were successfully migrated in the first task are counted as skipped files in the new task.
- Errors: The number of failed files. This is displayed only when migration failures occur.
- Elapsed time: The total migration duration.
- ScanComplete: Indicates whether the scan is complete (true means complete, false means not complete).

Migrated directory log

The job_name-folder.log file records uploaded directories. This file is generated only if transfer_dir is set to true in the task configuration file. It adds an entry for each successfully uploaded directory. The file records the following information:
- Column 1: Directory path name.

Migration failure log

The job_name-transfers-error.log file records files that failed to migrate. The log is updated in real time until the task completes. The file records the following information:
- Column 1: File path name.
- Column 2: Migration failure information.

Successful migration CRC-64 list log

The job_name-crc64-succeed.log file records the list of files that migrated successfully, along with their CRC-64 values. The log is updated in real time until the task completes. The file records the following information:
- Column 1: File CRC-64 value.
- Column 2: File path name.

Failed migration CRC-64 list log

The job_name-crc64-error.log file records the list of files that failed CRC-64 verification. The log is updated in real time until the task completes. The file records the following information:
- Column 1: File CRC-64 value.
- Column 2: File path name.

Skipped migration log

The job_name-crc64-skipped.log file records files that were skipped during the task. The log is updated in real time until the task completes. The file records the following information:
- Column 1: File path name.