Use DataWorks and a DistCp node for large-scale file migration

Parameter

Description

Example

path

Specifies the source directories from which data is read. Data is read from the source directories in order of the directory hierarchy and written to the corresponding destination file system.

Format: A string array, indicating that multiple source directories can be specified.

Default value: None.

"path": [ "/tmp/app_data/" ]

pattern

Specifies a file filter for data reads. Under the source directories specified in path, only files that pass the filter are transferred to the destination. The pattern filter applies to each source path directory. You can control which files should be included or excluded in the copy operation.

Format: A string array. Each filter element is prefixed with + for include or - for exclude. The + or - sign is followed by a space, and then the filter rule (which can be a regular expression).

Default value: Empty, indicating that no filter is applied and all files are transferred.

"pattern": [ "+ a.txt", "+ c/d.txt", "- *.log"]

• This example includes a.txt, includes d.txt in subdirectory c, and excludes all .log files.

• If both an include rule and an exclude rule match the same file, the include rule takes precedence.

• Files that are not matched by any include or exclude rule are transferred.

splitMode

Specifies the data synchronization strategy, which primarily affects the sharding strategy on the reader side. The following sharding strategies are supported:

• dynamic: A dynamic strategy in which each concurrent thread can transfer a different amount of data. Faster threads transfer more bytes.

• uniformsize: A uniform-size strategy in which the task is split so that each concurrent thread synchronizes files of approximately the same total byte size.

Default value: uniformsize.

"splitMode": "dynamic"

listFileConcurrent

Specifies the concurrency for listing files to be synchronized. A higher concurrency speeds up file discovery.

Default value: 1. The maximum value is 40. Values greater than 40 are automatically set to 40.

"listFileConcurrent": 10

Parameter

Description

Example

path

Specifies the destination directory to which data is written. Data from the source is written to the destination path following the directory hierarchy.

Format: A string.

Default value: None.

"path": "/tmp/app_data/"

enableRelative

Controls the directory behavior for data synchronization writes.

• true: Recreates the full directory hierarchy of the source file under the destination directory.

  Example: Source hdfs://hdfs:port/a/b/c/d.txt, destination oss://oss:port/. After synchronization, the file is written to oss://oss:port/a/b/c/d.txt.

• false: Uses the destination directory as the root and does not recreate the source directory structure. Only files are synchronized.

  Example: Source hdfs://hdfs:port/a/b/c/d.txt, destination oss://oss:port/. After synchronization, the file is written to oss://oss:port/d.txt.

Default value: false.

"enableRelative": "true"

writeMode

Specifies the data write mode. Three write modes are supported: overwrite, update, and append. The following list describes each mode:

• overwrite: Forcibly overwrites existing files in the destination path. If the destination path already contains files with the same names as files in the source path, those files are overwritten.

• update: Updates only files that are older or do not exist in the destination path. This mode skips files that are already up-to-date in the destination path, which saves time and bandwidth.

• append: Appends source file content to the destination files instead of replacing them. This mode is primarily used for log files and other files that require data accumulation.

  Note

  In append mode, skipCrcCheck is forcibly set to true.

Default value: None. This parameter is required.

"writeMode": "update"

preserveStatus

Controls whether file metadata is synchronized during file transfer. Valid values:

• REPLICATION

• BLOCKSIZE

• USER

• GROUP

• PERMISSION

• CHECKSUMTYPE

• ACL

• XATTR

• TIMES

• OWNER

• REPLICATION FACTOR

Format: A string array, indicating that multiple metadata attributes can be synchronized.

Default value: Empty, indicating that no metadata is synchronized.

"preserveStatus": ["ACL"]

chunkSize

Specifies whether to split a file into chunks during transfer and merge the content at the destination. This feature is available only when the source file system supports the getBlockLocations method and the destination file system supports the concat method. The value specifies the number of blocks per file. The byte size of each block is an HDFS system configuration, such as 64 MB.

Default value: 0, indicating no splitting.

"chunkSize": 0

bufferSize

Specifies the data buffer size, which affects the efficiency of data writes.

Default value: 8×1024, which is 8 KB.

"bufferSize": 8192

deleteMissingSource

Specifies whether to delete files and directories that exist in the destination but not in the source.

Valid values:

• true: Deletes extra files and directories in the destination.

• false: Does not delete extra files and directories in the destination.

Default value: false.

"deleteMissingSource": "false"

hadoopConfig

You can specify advanced configurations in hadoopConfig for accessing the destination HDFS, such as the access endpoints for OSS and OSS-HDFS:

• OSS configuration example:

  {
      "fs.oss.endpoint": "oss-cn-shanghai-internal.aliyuncs.com"
  }

• OSS-HDFS configuration example:

  {
      "fs.oss.endpoint": "cn-shanghai.oss-dls.aliyuncs.com",
      "fs.AbstractFileSystem.oss.impl": "com.aliyun.jindodata.oss.OSS",
      "fs.oss.impl": "com.aliyun.jindodata.oss.JindoOssFileSystem",
      "fs.oss.credentials.provider": "com.aliyun.jindodata.auth.SimpleAliyunCredentialsProvider"
  }

• If accessing OSS or OSS-HDFS requires a specific AccessKey pair, configure the following:

  {
      "fs.oss.endpoint": "oss-cn-shanghai-internal.aliyuncs.com",
      "fs.oss.accessKeyId": "xxxxx",
      "fs.oss.accessKeySecret": "xxxxx"
  }

Parameter

Description

Example

skipCrcCheck

Specifies whether to skip CRC verification between the source and destination.

Valid values:

• true: Skips CRC verification.

• false: Does not skip CRC verification.

Default value: false.

"skipCrcCheck": "false"

checksumCombineMode

When CRC verification is enabled, an error may occur if the source and destination are heterogeneous systems:

Their checksum algorithms may be incompatible You can choose file-level checksum validation via -Ddfs.checksum.combine.mode=COMPOSITE_CRC when block-sizes or filesystems are different. Or you can skip checksum-checks altogether  with -skipcrccheck.
 (NOTE: By skipping checksums, one runs the risk of masking data-corruption during file-transfer.)

You can set "skipCrcCheck": "true" to skip CRC verification.

Alternatively, you can use this parameter to specify the checksum generation and merge strategy between the source and destination storage systems during DistCp file copy. Valid values:

• MD5MD5CRC: Default behavior, which uses a dual MD5+CRC checksum method.

• COMPOSITE_CRC: A composite cyclic redundancy check (CRC) method. Testing has shown that this method can pass destination-side verification in HDFS-to-OSS-HDFS scenarios.

"checksumCombineMode": "COMPOSITE_CRC"

ignoreFailures

Specifies whether to ignore errors during synchronization, allowing the task to continue even if individual files fail to synchronize.

Valid values:

• true: The task continues even if individual files fail to synchronize.

• false: The task is interrupted if a file fails to synchronize.

Default value: false.

"ignoreFailures": "true"

concurrent

Specifies the concurrency of the transfer task. At most this number of files can be in the transferring state simultaneously. Under the hood, concurrent is the maximum parallelism of Map tasks in the MapReduce program.

Default value: 10.

"concurrent": 10

mbps

Controls the maximum transfer speed per concurrent thread, in MB/s. The synchronization program ensures that each concurrent thread does not exceed this limit. The overall speed limit for the task is mbps × concurrent.

Valid values:

• A value ≤ 0 means no throttling.

• A value > 0 means throttling is enabled. You can specify a floating-point number, for example, 5.5, which means the speed limit is 5.5MB/s.

Default value: -1.

"mbps": 10

queue

Specifies the queue in which the task runs. The underlying file synchronization task is a MapReduce program. You can use the queue parameter to specify the YARN queue for the task, which helps monitor and limit resource usage.

Default value: None.

"queue": "distcpQueue"