DocsICP FilingConsole
官方文档
首页

CreateDataFlowTask

更新时间：
复制 MD 格式
我的收藏

Create a data flow task.

Operation description

  • CPFS usage notes

    • Data flow is supported only on CPFS 2.2.0 and later. You can view the version information on the file system details page in the console.

    • Data flow tasks execute asynchronously. You can query the task status by calling the DescribeDataFlowTasks operation. Task duration depends on the amount of data. For large datasets, split the workload into multiple tasks.

    • You can create data flow tasks only on a data flow that is in the Running state.

    • Manually running a data flow task pauses the corresponding automatic data update task.

    • When you create an export task, ensure that the absolute path of each file to be exported from CPFS does not exceed 1,023 characters.

  • CPFS AI-Computing Edition usage notes

    • Data flow is supported only on CPFS AI-Computing Edition 2.4.0 and later. You can view the version information on the file system details page in the console.

    • Data flow tasks execute asynchronously. You can query the task status by calling the DescribeDataFlowTasks operation. Task duration depends on the amount of data. For large datasets, split the workload into multiple tasks.

    • You can create data flow tasks only on a data flow that is in the Running state.

    • When you create an export task, ensure that the absolute path of each file to be exported from CPFS AI-Computing Edition does not exceed 1,023 characters.

    • CPFS AI-Computing Edition supports two task types: batch tasks and streaming tasks. For more information, see Task types.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

  • Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.

  • API: The API that you can call to perform the action.

  • Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.

  • Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.

    • For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.

    • For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.

  • Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.

  • Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

nas:CreateDataFlowTask

create

*DataFlow

acs:nas:{#regionId}:{#accountId}:filesystem/{#filesystemId}

 None None

Request parameters

Parameter

Type

Required

Description

Example
FileSystemId

string

Yes

The ID of the file system.

  • CPFS General Purpose Edition: The ID must start with cpfs-, such as cpfs-125487****.

  • CPFS AI-Computing Edition: The ID must start with bmcpfs-, such as bmcpfs-0015****.

cpfs-099394bd928c****
DataFlowId

string

Yes

The ID of the data flow.

df-194433a5be31****
SrcTaskId

string

No

If you specify SrcTaskId, enter the ID of a data flow task. The system copies the TaskAction, DataType, and EntryList parameter information from the specified task. You do not need to specify these parameters.

Note

Data flow streaming tasks are not supported.

task-29ee8e890f45****
TaskAction

string

No

The type of the data flow task.

Valid values:

  • Import: Imports specified data from the source storage to the CPFS file system.

  • Export: Exports specified data from the CPFS file system to the source storage.

  • StreamImport: Imports a large amount of specified data from the source storage to the CPFS file system.

  • StreamExport: Exports a large amount of specified data from the CPFS file system to the source storage.

  • Evict: Releases the data blocks of a file from the CPFS file system. After the release, only the metadata of the file is retained. You can still query the file, but its data blocks are cleared and no longer occupy storage capacity. When you access the file data, it is loaded from the source storage on demand.

  • Inventory: Obtains the inventory of files managed by a data flow for the CPFS file system. This provides the cache status of files in the data flow.

Note

CPFS AI-Computing Edition supports only Import, Export, StreamImport, and StreamExport. StreamImport and StreamExport are supported only by CPFS AI-Computing Edition V2.6.0 and later.

Import
DataType

string

No

The data type that the data flow task operates on.

Valid values:

  • Metadata: The metadata of the file, including attributes such as timestamp, ownership, and permissions. If you select Metadata, only the file metadata is imported. You can see the file, but when you access the file data, it is loaded from the source storage on demand.

  • Data: The data blocks of the file.

  • MetaAndData: The metadata and data blocks of the file.

Metadata
Directory

string

No

The source directory of the data.

Limits:

  • The length must be 1 to 1,023 characters.

  • The directory must be UTF-8 encoded.

  • The directory must start and end with a forward slash (/).

  • Only one directory can be specified at a time.

  • If TaskAction is Export, this directory must be a relative path within FileSystemPath.

  • If TaskAction is Import, this directory must be a relative path within SourceStoragePath.

  • If TaskAction is StreamExport, this directory must be a relative path within FileSystemPath.

  • If TaskAction is StreamImport, this directory must be a relative path within SourceStoragePath.

Note

StreamImport and StreamExport are supported only by CPFS AI-Computing Edition V2.6.0 and later.

/path_in_cpfs/
EntryList

string

No

The list of files for the data flow task to execute.

Limits:

  • The list must be UTF-8 encoded.

  • The total length of the file list must be less than 64 KB.

  • The file list must be in JSON format.

  • The path of a single file must be 1 to 1,023 characters in length and must start with a forward slash (/).

  • If TaskAction is Import, each element in the list represents an OSS Object name.

  • If TaskAction is Export, each element in the list represents a CPFS file path.

["/path_in_cpfs/file1", "/path_in_cpfs/file2"]
DryRun

boolean

No

Specifies whether to perform a dry run for this creation request.

A dry run checks parameter validity and inventory without creating an instance or incurring charges.

Valid values:

  • true: Sends a check request without creating the instance. The system checks for required parameters, request format, business limits, and NAS inventory. If the check fails, an error is returned. If the check passes, an HTTP 200 status code is returned, but TaskId is empty.

  • false (default): Sends a normal request and creates the instance after the check passes.

false
ClientToken

string

No

A client-generated token that ensures the idempotence of the request. The token must be unique across different requests.

ClientToken can contain only ASCII characters and must not exceed 64 characters. For more information, see How to ensure idempotence.

Note

If you do not specify this parameter, the system automatically uses the RequestId of the API request as the ClientToken. The RequestId may be different for each API request.

123e4567-e89b-12d3-a456-42665544****
ConflictPolicy

string

No

The conflict policy for files with the same name. Valid values:

  • SKIP_THE_FILE: Skips files with the same name.

  • KEEP_LATEST: Compares update times and keeps the latest version.

  • OVERWRITE_EXISTING: Forcibly overwrites files with the same name.

Note

This parameter is required if the file system is a CPFS AI-Computing Edition instance.

SKIP_THE_FILE
DstDirectory

string

No

The destination directory for the data flow task mapping. Limits:

  • The directory must start and end with a forward slash (/). The /../ sequence is not supported.

  • The length must be 1 to 1,023 characters.

  • The directory must be UTF-8 encoded.

  • Only one directory can be specified at a time.

  • If TaskAction is Export, this directory must be a relative path within SourceStoragePath.

  • If TaskAction is Import, this directory must be a relative path within FileSystemPath.

  • If TaskAction is StreamExport, this directory must be a relative path within SourceStoragePath.

  • If TaskAction is StreamImport, this directory must be a relative path within FileSystemPath.

Note

StreamImport and StreamExport are supported only by CPFS AI-Computing Edition V2.6.0 and later.

/path_in_cpfs/
CreateDirIfNotExist

boolean

No

Specifies whether to automatically create the directory if it does not exist. Valid values:

  • true: Automatically creates the directory.

  • false (default): Does not automatically create the directory.

Note

  • This parameter takes effect only when TaskAction is set to Import.

  • This parameter is supported only by CPFS AI-Computing Edition V2.6.0 and later.

false
Includes

string

No

Filters the subdirectories under the Directory parameter and transfers the content of the filtered subdirectories.

Note

  • This parameter takes effect only when the Directory parameter is specified.

  • The path of a single folder must be 1 to 1,023 characters in length and must start and end with a forward slash (/). The total length cannot exceed 3,000 characters.

  • This feature is supported only by CPFS AI-Computing Edition.

["/test/","/test1/"]
TransferFileListPath

string

No

Specifies an OSS directory. Data is synchronized based on the content of the CSV files in this directory. The following limits apply.

  • The path must start and end with a forward slash (/).

  • The path is case-sensitive.

  • The length must be between 1 and 1,023 characters.

  • The path must be UTF-8 encoded.

Note

  • TransferFileListPath, Directory, and EntryList are mutually exclusive. You can specify only one of them.

  • This parameter specifies an existing path in OSS. The *.csv files are stored in this path.

  • TransferFileListPath supports only the Import and Export features.

  • For an Import task, the files or directories specified in the CSV file are imported from OSS to the CPFS file system.

  • For an Export task, the files or directories specified in the CSV file are exported from the CPFS file system to OSS.

  • The CSV file must contain Name and Type columns. Name is the relative path. Type can be dir or file. If Type is dir, the Name value must end with a forward slash (/).

  • This feature is supported only by CPFS AI-Computing Edition.

/test_oss_path/

Response elements

Element

Type

Description

Example

object

RequestId

string

The request ID.

2D69A58F-345C-4FDE-88E4-BF518948****
TaskId

string

The Dataflow task ID.

task-38aa8e890f45****

Examples

Success response

JSON format

{
  "RequestId": "2D69A58F-345C-4FDE-88E4-BF518948****",
  "TaskId": "task-38aa8e890f45****"
}

Error codes

HTTP status code

Error code

Error message

Description
400 IllegalCharacters The parameter contains illegal characters. The parameter contains illegal characters.
400 MissingFileSystemId FileSystemId is mandatory for this action.
400 MissingDataFlowId DataFlowId is mandatory for this action.
400 InvalidFilesystemVersion.NotSupport This Api does not support this fileSystem version. This Api does not support this fileSystem version.
403 OperationDenied.InvalidState The operation is not permitted when the status is processing. The operation is not permitted when the status is processing.
403 OperationDenied.DependencyViolation The operation is denied due to dependancy violation.
403 OperationDenied.DataFlowNotSupported The operation is not supported.
404 InvalidParameter.InvalidFormat The EntryList format is invalid.
404 InvalidParameter.SizeTooLarge The specified EntryList size exceeds 64 KB.
404 InvalidDataFlow.NotFound The specified data flow does not exist.
404 InvalidTaskAction.NotSupported The task action is not supported.
404 InvalidTaskAction.PermissionDenied The task action is not allowed.
404 InvalidSrcTaskId.NotFound The SrcTaskId is not found.
404 InvalidDataType.NotSupported The data type is not supported.
404 InvalidSrcTaskId.TaskIdInvalid Source task ID is invalid.
404 InvalidSrcTaskId.TaskIdNotFound Source task ID is not found.

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.