Copy local data-Data Transport(Data Transport)-阿里云帮助中心

This topic describes how to create and execute data migration tasks using Data Transport II, and how to view task logs to read data from the data source and store it in Data Transport.

Important notes

When migrating data with Data Transport, note the following:

For the source path Directory To Be Migrated, use an absolute path that starts and ends with a forward slash (/). Environment variables and special characters are not supported. Ensure the Directory To Be Migrated exists and is valid.
Data Transport uses resources from the source path during migration, which may affect normal business operations. If your business is critical, evaluate the impact beforehand, set rate limits on the source, or start the migration task during off-peak hours.
If files with identical names exist in both the source path and the destination OSS bucket, Data Transport overwrites the destination files by default. If the file contents differ, rename the files or back them up before migration.

By default, Data Transport preserves the last modification time of source files. If the destination bucket has a lifecycle rule enabled and migrated files fall within the rule’s effective period, those files may be deleted or transitioned to a specified archive storage class when the rule takes effect.
- If you do not need to preserve the last modification time, contact Alibaba Cloud support to adjust this setting. Confirm that you fully understand the associated risks before proceeding.

Migration limitations

Character device files, block device files, socket files, and named pipe files in the source path will not be migrated.
If the source path contains symbolic links, they are migrated only if transfer_symlink is set to true. Otherwise, they are skipped. For more information about symbolic links, see Symbolic links.
If hard links exist in the source path, they become regular files after migration.
Parent directory attributes are not migrated.
File permissions such as SUID, SGID, and SBID are not supported for migration.
The combined length of a directory path and filename must not exceed 1,024 bytes. Each directory name must be no longer than 200 bytes, and each filename must be no longer than 255 bytes. Files exceeding these limits will not be migrated.
Supported attributes for migration: Permissions → X-Oss-Meta-Perms (OSS), Uid:Gid → X-Oss-Meta-Owner (OSS), mtime → X-Oss-Meta-Mtime (OSS), atime → X-Oss-Meta-Atime (OSS), ctime → X-Oss-Meta-Ctime (OSS).
Note
- Permissions: Includes nine standard read, write, and execute permissions.
- Uid:Gid: User ID:Group ID, separated by a colon (:).
Unsupported attributes (including but not limited to): AccessTime, Attr, Acl.

Note
The list above is not exhaustive. Behavior for other unlisted attributes is undetermined. Refer to the actual migration results.

Preparations

1. Decrypt and mount the storage pool

After powering on, the storage pool is locked. Decrypt it using the initial password or add a new password. To obtain the initial password, contact Alibaba Cloud support.

To decrypt the storage pool using the initial password:
- Run crypt open <password>.
Add a new key and decrypt the storage pool as follows:
- Run crypt addsecret <old_password> <new_password>.
- To delete the initial password, run crypt deletesecret <old_password>.
- To open the storage pool with the new password, run crypt open <new_password>.

Note

When setting a new password, include uppercase letters, lowercase letters, numbers, and special symbols. Use at least 8 characters. Example: Hello@12345!.

2. Initialize mgwserver

Important

Data Transport currently supports only one bucket. To create multiple buckets, contact Alibaba Cloud support.

Create a bucket, access key (AK), and secret key (SK) for this migration.

Check whether the mgwserver service status is normal by running mgwserver status. If the status is abnormal, contact Alibaba Cloud support.
Create a bucket and set the AK and SK by running mgwserver create-bucket <Bucket> <AK> <SK>.
Note
- Bucket: Bucket name. Use the same name as your OSS bucket in the cloud, for example, xxx-xxx-data.
- AK: Username. Use lowercase letters, numbers, hyphens (-), and underscores (_). Example: aliyuncs@2024-xx-xx.
- SK: Password. Use lowercase letters, numbers, hyphens (-), and underscores (_). Example: xxxxxxxxx.

Create a migration task

Important

Specify tasks at the directory level. To migrate all data from the source, create one task. To migrate data from N directories, create N tasks. Determine the number of tasks needed and copy the corresponding number of job.cfg files—one per task.

Configure the task file

Run the command cdmgwclient && cd conf to switch to the specified folder.
Determine the number of tasks to create and copy that many demo-job.properties files. Each task requires one demo-job.properties file.
Note
- Rename each demo-job.properties file to distinguish them, for example: job1.properties, job2.properties.
- If migrating an entire directory, create only one task.
- If migrating specific subdirectories, create one task per directory. The following steps use job1.properties as an example.

Open one properties file, such as job1.properties, and configure the following parameters:

Parameter	Required	Example value
job_name	Yes	Task name, for example: example_job.
src_path	Yes	Source path. Must start and end with a forward slash (/). Example: If the source path is `/mnt/nas/example_dir/` and it contains example.jpg, and the destination path is `mgw:example_bucket/dest/`, then after migration, the full path of example.jpg becomes `mgw:example_bucket/dest/example.jpg`.
dest_path	Yes	Destination path in Data Transport, formatted as `mgw:BucketName/prefix/`. Example: `mgw:example_bucket/dest/`. BucketName: The bucket name configured during mgwserver initialization. prefix: Specifies a prefix under the bucket. Must end with a forward slash (/). With prefix: If the source path is `/mnt/nas/example_dir/` containing example.jpg, and the bucket prefix is `example/dest/`, then after migration, the full path becomes `mgw:example_bucket/example/dest/example.jpg`. Without prefix: If the source path is `/mnt/nas/example_dir/` containing example.jpg, and no prefix is set, then after migration, the full path becomes `mgw:example_bucket/example.jpg`.
transfer_dir	Yes	Migrate directories. Set to true to enable, false to disable. When enabled, all directories found in the source path are added to the migration queue and logged. Empty objects ending with '/' are created in the destination OSS bucket, and directory attributes (if supported) are stored in the object’s UserMeta. When disabled, directories are ignored and excluded from logs. No empty '/' objects are created in the destination OSS bucket.
transfer_symlink	Yes	Migrate symbolic links. Set to true to enable, false to disable. When enabled, symbolic link files are added to the migration queue and counted in the task’s file count and data volume. Symlink objects are created in the destination OSS bucket, and symlink attributes (if supported) are stored in the object’s UserMeta. To determine the Target property of the Symlink object: first resolve the symlink’s target to its shortest absolute path (AbsTarget) relative to the symlink’s directory, then replace the src_path portion of AbsTarget (if matched) with the dest_path prefix, and assign the result to the Symlink object’s Target property. Note Example: Assume src_path="/mnt/nas/example_dir/", dest_path prefix="example/dest/", and /mnt/nas/example_dir/links/a.lnk is a symbolic link with Target: "../data/./a.txt" → resolved to "/mnt/nas/example_dir/data/a.txt" → final Target: "example/dest/data/a.txt". "/mnt/nas/example_dir/verbose/../data/./a.txt" → resolved to "/mnt/nas/example_dir/data/a.txt" → final Target: "example/dest/data/a.txt". When disabled, symbolic links are ignored and excluded from file count and data volume metrics. Important In all cases, target files or directories pointed to by symbolic links are not migrated unless they are also within the current migration scope.
transfers	No	Concurrency per task. Default is 16. Keep the default for large-file migrations. For small-file migrations, set to 256 (maximum).
log_level	No	Log level. Default is INFO. Set to DEBUG for more detailed logs.

Start the task

Important

Note: Run cdmgwclient to enter the directory before performing the following operations.

Run bash console.sh start conf/properties filename to start the task. Example: bash console.sh start conf/job1.properties.

View task status

Important

Note that you must run commands involving bash console.sh after navigating to the cdmgwclient directory.

To view task status:

Run bash console.sh status <job_name> to check task status. (Replace <job_name> with your actual task name.)

If JobStatus is running, the task is still migrating. You can view the current progress:

JobStatus:        running
Transferred:        2.719 GiB / 2.878 GiB, 94%, 13.238 MiB/s, ETA 12s
Transferred:          2270 / 9307, 24%
ScanComplete:       true
 * xxx

If JobStatus is failed, some files failed to migrate. Review logs to troubleshoot or contact Data Transport support for assistance.

JobStatus:       failed
Transferred:     29.270 KiB / 29.270 KiB, 100%, 0 B/s, ETA -
Transferred:          14 / 14, 100%
Checks:              10 / 10, 100%
Errors:               1 (retrying may help)
ScanComplete:    true
Elapsed time:    0.1s
you can view error message based on /home/aliyuncs/log/mgwclient/xxx-rt.log
you can view transfers error message based on /home/aliyuncs/log/mgwclient/xxx-error.log

If JobStatus is succeed, all files migrated successfully. View total data volume and file count:

JobStatus:        succeed
Transferred:      126.335 GiB / 126.335 GiB, 100%, 4.462 MiB/s, ETA 0s
Transferred:        9303 / 9303, 100%

Note

Parameter descriptions:

First Transferred: Scanned data volume / Successfully transferred data volume, shown as percentage, average transfer speed.
Second Transferred: Scanned file count / Successfully transferred file count, shown as percentage.
Checks: Skipped file count. Displayed only if files were skipped. For example, files successfully migrated in a prior attempt are counted as skipped during a retry.
Errors: Failed file count. Displayed only if migration failures occurred.
Elapsed time: Total migration duration.
ScanComplete: Whether scanning finished (true = complete, false = incomplete).

Important

If the source data consists of a large flat directory structure, scanning takes longer and progress updates appear slower.

Delete a task

If you started an incorrect task, delete it and its log directory using:

Run bash console.sh delete <job_name> to delete the specified task. (Replace <job_name> with the actual task name. Already migrated files are not deleted.)

Post-migration steps

Important

All following operations are performed in the /home/aliyuncs/log/mgwclient/job_name directory.

If the task status is failed, review logs to identify the cause or contact Data Transport support for assistance.
If the task status is succeed, verify source files against the CRC64 list. Always perform data validation.

View logs

Important

Replace job_name in the following paths with your actual task name.

Logs recording file upload status and task status are stored in /home/aliyuncs/log/mgwclient/job_name.

Log Name	Log File Description
job_name-status.log	Task migration progress.
job_name-folder.log	List of migrated directories.
job_name-transfers-error.log	List of files that failed to migrate.
job_name-crc64-succeed.log	CRC64 list of successfully migrated files.
job_name-crc64-error.log	List of files that failed CRC64 validation during migration.
job_name-crc64-skipped.log	List of skipped files during migration.

Important

Migration failures fall into two categories: transfer failures and CRC64 validation failures. The total failure count combines both.

Task progress log

job_name-status.log records basic task progress.
- First Transferred: Scanned data volume / Successfully transferred data volume, shown as percentage, average transfer speed.
- Second Transferred: Scanned file count / Successfully transferred file count, shown as percentage.
- Checks: Skipped file count. Displayed only if files were skipped. For example, files successfully migrated in a prior attempt are counted as skipped during a retry.
- Errors: Failed file count. Displayed only if migration failures occurred.
- Elapsed time: Total migration duration.
- ScanComplete: Whether scanning finished (true = complete, false = incomplete).

Migrated directories log

job_name-folder.log logs uploaded directories. Only generated if transfer_dir=true in the task configuration. Each successfully uploaded directory adds one log entry containing:
- Column 1: Directory path.

Migration failure log

job_name-transfers-error.log logs files that failed to migrate. Updated in real time until the task ends. Each entry contains:
- Column 1: File path.
- Column 2: Migration failure reason.

Migrated CRC64 success log

job_name-crc64-succeed.log logs CRC64 values of successfully migrated files. Updated in real time until the task ends. Each entry contains:
- Column 1: File CRC64 value.
- Column 2: File path.

Failed migration CRC-64 list log

job_name-crc64-error.log logs CRC64 values of files that failed validation. Updated in real time until the task ends. Each entry contains:
- Column 1: File CRC64 value.
- Column 2: File path.

Skipped files log

job_name-crc64-skipped.log logs skipped files. Updated in real time until the task ends. Each entry contains:
- Column 1: File path.