DocsICP FilingConsole
官方文档
首页

CreateDataFlowTask

更新时间：
复制 MD 格式
我的收藏

Create a data flow task.

Operation description

  • Notes on CPFS

    • Data flow is available only for CPFS 2.2.0 and later. You can check the file system version on the File System Details page in the console.

    • Data flow tasks are asynchronous. You can call the DescribeDataFlowTasks operation to query the execution status of a task. The task duration depends on the amount of data to be imported or exported. For large amounts of data, we recommend splitting it into multiple tasks.

    • You can create data flow tasks only for data flows in the Running state.

    • Manually running a data flow task suspends the automatic update task for that data flow.

    • When creating an export task, ensure that the total length of the absolute path for any exported file from CPFS does not exceed 1023 characters.

  • Notes on CPFS for AI

    • Data flow is available only for CPFS for AI 2.4.0 and later. You can check the file system version on the File System Details page in the console.

    • Data flow tasks are asynchronous. You can call the DescribeDataFlowTasks operation to query the execution status of a task. The task duration depends on the amount of data to be imported or exported. For large amounts of data, we recommend splitting it into multiple tasks.

    • You can create data flow tasks only for data flows in the Running state.

    • When creating an export task, ensure that the total length of the absolute path for any exported file from CPFS for AI does not exceed 1023 characters.

    • CPFS for AI supports two task types: batch task and streaming task. For more information, see Introduction to task types.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

  • Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.

  • API: The API that you can call to perform the action.

  • Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.

  • Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.

    • For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.

    • For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.

  • Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.

  • Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

nas:CreateDataFlowTask

create

*DataFlow

acs:nas:{#regionId}:{#accountId}:filesystem/{#filesystemId}

 None None

Request parameters

Parameter

Type

Required

Description

Example
FileSystemId

string

Yes

The ID of the file system.

  • CPFS General-purpose Edition: The ID must start with cpfs-. Example: cpfs-125487****.

  • CPFS AI Computing Edition: The ID must start with bmcpfs-. Example: bmcpfs-0015****.

cpfs-099394bd928c****
DataFlowId

string

Yes

The data flow ID.

df-194433a5be31****
SrcTaskId

string

No

The ID of a source data flow task. If you specify this parameter, the system copies the TaskAction, DataType, and EntryList parameter values from the specified source task. You do not need to configure these parameters in your request.

Note

This parameter does not support streaming data flow tasks.

task-29ee8e890f45****
TaskAction

string

No

The type of the data flow task.

Valid values:

  • Import: Imports the specified data from the source storage to CPFS.

  • Export: Exports the specified data from CPFS to the source storage.

  • StreamImport: Imports data in batches from the source storage to CPFS.

  • StreamExport: Exports data in batches from CPFS to the source storage.

  • Evict: Evicts the data blocks of files in the CPFS file system. After the task is complete, only the file metadata is retained in CPFS. You can still query the file, but its data blocks are cleared and no longer consume storage capacity. Accessing the file again loads its data from the source storage on demand.

  • Inventory: Retrieves an inventory of files managed by the data flow, including their cache status.

Note

CPFS AI Computing Edition supports only Import, Export, StreamImport, and StreamExport. The StreamImport and StreamExport actions are supported only on CPFS AI Computing Edition 2.6.0 and later.

Import
DataType

string

No

The type of data to be processed by the data flow task.

Valid values:

  • Metadata: The metadata of a file, such as its timestamp, ownership, and permissions. If you select this value, only the file metadata is imported. You can see the file in the file system, but its data is loaded on demand from the source storage when accessed.

  • Data: The data blocks of a file.

  • MetaAndData: The metadata and data blocks of a file.

Metadata
Directory

string

No

The path of the source directory for the data flow task.

Constraints:

  • The value must be 1 to 1,023 characters in length.

  • The value must be encoded in UTF-8.

  • The value must start and end with a forward slash (/).

  • You can specify only one directory at a time.

  • If TaskAction is set to Export, the directory must be a relative path in FileSystemPath.

  • If TaskAction is set to Import, the directory must be a relative path in SourceStoragePath.

  • If TaskAction is set to StreamExport, the directory must be a relative path in FileSystemPath.

  • If TaskAction is set to StreamImport, the directory must be a relative path in SourceStoragePath.

Note

The StreamImport and StreamExport actions are supported only on CPFS AI Computing Edition 2.6.0 and later.

/path_in_cpfs/
EntryList

string

No

A list of files for the data flow task to process.

Constraints:

  • The value must be encoded in UTF-8.

  • The total length of the file list cannot exceed 64 KB.

  • The file list must be in JSON format.

  • Each file path must be 1 to 1,023 characters in length and must start with a forward slash (/).

  • If TaskAction is Import, each element is the path to an OSS object.

  • If TaskAction is Export, each element is the path to a CPFS file.

["/path_in_cpfs/file1", "/path_in_cpfs/file2"]
DryRun

boolean

No

Specifies whether to perform a dry run for the request.

A dry run checks for parameter validity and resource availability without creating the task or incurring charges.

Valid values:

  • true: Sends a check request without creating the task. The system checks for required parameters, request format, business limits, and NAS inventory. If the check fails, an error is returned. If the check succeeds, an HTTP 200 status code is returned, but the TaskId is empty.

  • false (default): Sends a normal request and creates the task after the check is passed.

false
ClientToken

string

No

A client-generated token that ensures the idempotence of the request. The token must be unique across requests.

ClientToken can contain only ASCII characters and cannot exceed 64 characters in length. For more information, see How to ensure idempotence.

Note

If you do not specify this parameter, the system automatically uses the RequestId of the API request as the ClientToken. The RequestId is different for each request.

123e4567-e89b-12d3-a456-42665544****
ConflictPolicy

string

No

The conflict policy for files with the same name. Valid values:

  • SKIP_THE_FILE: Skips the file.

  • KEEP_LATEST: Compares the update times and keeps the latest version.

  • OVERWRITE_EXISTING: Overwrites the existing file.

Note

This parameter is required for CPFS AI Computing Edition file systems.

SKIP_THE_FILE
DstDirectory

string

No

The destination directory for the data flow task. Constraints:

  • The value must start and end with a forward slash (/). The ../ sequence is not supported.

  • The value must be 1 to 1,023 characters in length.

  • The value must be encoded in UTF-8.

  • You can specify only one directory at a time.

  • If TaskAction is set to Export, the directory must be a relative path in SourceStoragePath.

  • If TaskAction is set to Import, the directory must be a relative path in FileSystemPath.

  • If TaskAction is set to StreamExport, the directory must be a relative path in SourceStoragePath.

  • If TaskAction is set to StreamImport, the directory must be a relative path in FileSystemPath.

Note

The StreamImport and StreamExport actions are supported only on CPFS AI Computing Edition 2.6.0 and later.

/path_in_cpfs/
CreateDirIfNotExist

boolean

No

Specifies whether to automatically create the destination directory if it does not exist. Valid values:

  • true: Automatically creates the directory.

  • false (default): Does not automatically create the directory.

Note

  • This parameter takes effect only when TaskAction is set to Import.

  • This feature is supported only on CPFS AI Computing Edition 2.6.0 and later.

false
Includes

string

No

A list of subdirectories under the Directory path to include in the transfer. If specified, only the content of these subdirectories is transferred.

Note

  • This parameter takes effect only when the Directory parameter is specified.

  • Each directory path must be 1 to 1,023 characters in length and must start and end with a forward slash (/). The total length of all paths cannot exceed 3,000 characters.

  • This feature is supported only for CPFS AI Computing Edition.

["/test/","/test1/"]
TransferFileListPath

string

No

Specifies an OSS directory containing CSV files that list the data to be synchronized. Constraints:

  • The value must start and end with a forward slash (/).

  • The value is case-sensitive.

  • The value must be 1 to 1,023 characters in length.

  • The value must be encoded in UTF-8.

Note

  • The TransferFileListPath, Directory, and EntryList parameters are mutually exclusive. You can specify only one of them.

  • This parameter must specify a path that exists in OSS. The *.csv files are stored in OSS at this path.

  • TransferFileListPath is supported only for Import and Export tasks.

  • For an Import task, the files or directories specified in the CSV file are imported from OSS to CPFS.

  • For an Export task, the files or directories specified in the CSV file are exported from CPFS to OSS.

  • The CSV file must contain Name and Type columns. Name specifies the relative path. Type can be dir or file. If Type is dir, the value in the Name column must end with a forward slash (/).

  • This feature is supported only for CPFS AI Computing Edition.

/test_oss_path/

Response elements

Element

Type

Description

Example

object

RequestId

string

The request ID.

2D69A58F-345C-4FDE-88E4-BF518948****
TaskId

string

The ID of the data flow task.

task-38aa8e890f45****

Examples

Success response

JSON format

{
  "RequestId": "2D69A58F-345C-4FDE-88E4-BF518948****",
  "TaskId": "task-38aa8e890f45****"
}

Error codes

HTTP status code

Error code

Error message

Description
400 IllegalCharacters The parameter contains illegal characters. The parameter contains illegal characters.
400 MissingFileSystemId FileSystemId is mandatory for this action.
400 MissingDataFlowId DataFlowId is mandatory for this action.
400 InvalidFilesystemVersion.NotSupport This Api does not support this fileSystem version. This Api does not support this fileSystem version.
403 OperationDenied.InvalidState The operation is not permitted when the status is processing. The operation is not permitted when the status is processing.
403 OperationDenied.DependencyViolation The operation is denied due to dependancy violation.
403 OperationDenied.DataFlowNotSupported The operation is not supported.
404 InvalidParameter.InvalidFormat The EntryList format is invalid.
404 InvalidParameter.SizeTooLarge The specified EntryList size exceeds 64 KB.
404 InvalidDataFlow.NotFound The specified data flow does not exist.
404 InvalidTaskAction.NotSupported The task action is not supported.
404 InvalidTaskAction.PermissionDenied The task action is not allowed.
404 InvalidSrcTaskId.NotFound The SrcTaskId is not found.
404 InvalidDataType.NotSupported The data type is not supported.
404 InvalidSrcTaskId.TaskIdInvalid Source task ID is invalid.
404 InvalidSrcTaskId.TaskIdNotFound Source task ID is not found.

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.