DocsICP FilingConsole
官方文档
首页

DescribeDataFlowTasks

更新时间：
复制 MD 格式
我的收藏

Retrieves data flow task details.

Operation description

Querying data flow tasks is supported only on CPFS 2.2.0 or later and CPFS AI Computing Edition 2.4.0 or later. You can find the version information on the file system details page in the console.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

  • Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.

  • API: The API that you can call to perform the action.

  • Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.

  • Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.

    • For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.

    • For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.

  • Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.

  • Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

nas:DescribeDataFlowTasks

get

*DataFlow

acs:nas:{#regionId}:{#accountId}:filesystem/{#filesystemId}

 None None

Request parameters

Parameter

Type

Required

Description

Example
FileSystemId

string

Yes

The file system ID.

  • CPFS General-purpose: The ID must start with cpfs-, such as cpfs-099394bd928c****.

  • CPFS for AI Computing: The ID must start with bmcpfs-, such as bmcpfs-290w65p03ok64ya****.

bmcpfs-290w65p03ok64ya****

Filters

array<object>

No

A collection of filters.

object

No

Key

string

No

The filter key.

Valid values:

  • DataFlowIds: Filters by data flow ID.

  • TaskIds: Filters by data flow task ID.

  • Originator: Filters by originator.

  • TaskActions: Filters by data flow task type.

  • DataTypes: Filters by data type.

  • Status: Filters by status.

  • CreateTimeBegin: Filters data flow tasks created after the specified time.

  • CreateTimeEnd: Filters data flow tasks created before the specified time.

  • StartTimeBegin: Filters data flow tasks that started after the specified time.

  • StartTimeEnd: Filters data flow tasks that started before the specified time.

  • EndTimeBegin: Filters data flow tasks that ended after the specified time.

  • EndTimeEnd: Filters data flow tasks that ended before the specified time.

DataFlowIds
Value

string

No

The filter value. This parameter does not support wildcards.

  • When Key is DataFlowIds, specify one or more data flow IDs. You can specify up to 10 data flow IDs, separated by commas. For example, df-194433a5be31**** or df-194433a512a2****,df-234533a5be31****.

  • When Key is TaskId, specify one or more data flow task IDs. You can specify up to 10 data flow task IDs, separated by commas. For example, task-38aa8e890f45**** or task-38aa8e890f45****,task-29ae8e890f45****.

  • When Key is TaskActions, specify the data flow task type. Valid values are Import, Export, Evict, Inventory, StreamImport, and StreamExport. You can specify multiple values. CPFS for AI Computing supports only Import, Export, StreamImport, and StreamExport. StreamImport and StreamExport are available only in CPFS for AI Computing 2.6.0 and later.

  • When Key is DataTypes, specify the data type of the data flow task. Valid values are MetaAndData, Metadata, and Data. You can specify multiple values.

  • When Key is Originator, specify the originator of the data flow task. Valid values are User and System.

  • When Key is Status, specify the status of the data flow task. Valid values are Pending, Executing, Failed, Completed, Canceling, and Canceled. You can specify multiple values.

  • When Key is CreateTimeBegin, specify the earliest creation time. Use the yyyy-MM-ddTHH:mmZ format.

  • When Key is CreateTimeEnd, specify the latest creation time. Use the yyyy-MM-ddTHH:mmZ format.

  • When Key is StartTimeBegin, specify the earliest start time. Use the yyyy-MM-ddTHH:mmZ format.

  • When Key is StartTimeEnd, specify the latest start time. Use the yyyy-MM-ddTHH:mmZ format.

  • When Key is EndTimeBegin, specify the earliest end time. Use the yyyy-MM-ddTHH:mmZ format.

  • When Key is EndTimeEnd, specify the latest end time. Use the yyyy-MM-ddTHH:mmZ format.

df-194433a5be31****
NextToken

string

No

The pagination token for the next page of results. If the response is truncated, use this token in your next request to retrieve the subsequent page.

TGlzdFJlc291cmNlU****mVzJjE1MTI2NjY4NzY5MTAzOTEmMiZORnI4NDhVeEtrUT0=
MaxResults

integer

No

The maximum number of results to return per page.

Valid values: 10 to 100.

Default value: 20.

20
WithReports

boolean

No

Specifies whether to return report information.

  • True (default): Includes reports in the response.

  • False: Excludes reports from the response.

Note

  • Set this parameter to False to speed up the query.

  • This parameter is supported only in CPFS for AI Computing.

True

Response elements

Element

Type

Description

Example

object

RequestId

string

The request ID.

2D69A58F-345C-4FDE-88E4-BF518948****
NextToken

string

If the response is truncated, use NextToken to retrieve the remaining results.

TGlzdFJlc291cmNlU****mVzJjE1MTI2NjY4NzY5MTAzOTEmMiZORnI4NDhVeEtrUT0=
TaskInfo

object

Task

array<object>

A list of data flow tasks.

array<object>

FilesystemId

string

The file system ID.

bmcpfs-290w65p03ok64ya****
DataFlowId

string

The data flow ID.

df-194433a5be31****
TaskId

string

The data flow task ID.

task-38aa8e890f45****
SourceStorage

string

The access path of the source storage. The format is <storage type>://[<account id>:]<path>.

Where:

  • storage type: The only supported type is OSS.

  • account id: The UID of the account that owns the source storage.

  • path: The name of the OSS bucket. The name must meet the following requirements:

    • It must contain only lowercase letters, digits, and hyphens (-), and must start and end with a lowercase letter or a digit.

    • Must be 128 characters or less.

    • It must be encoded in UTF-8.

Note

  • The OSS bucket must exist in the current region.

  • The account id parameter is supported only by CPFS computing edition 2.6.0 and later.

oss://178321033379****:bucket-01
FileSystemPath

string

The path of the fileset in the CPFS file system.

Limits:

  • The path must be 2 to 1,024 characters long.

  • The path must be encoded in UTF-8.

  • The path must start and end with a forward slash (/).

  • The path must be a fileset directory in the CPFS file system.

Note

This parameter is supported only by CPFS.

/a/b/c/
Originator

string

The originator of the data flow task. Valid values:

  • User: The task is initiated by a user.

  • System: The task is an automatic update triggered by the CPFS system at a set interval.

Note

This parameter is supported only by CPFS.

User
TaskAction

string

The type of the data flow task. Valid values:

  • Import: Imports data from the source storage to CPFS.

  • Export: Exports data from CPFS to the source storage.

  • StreamImport: Imports data from the source storage to CPFS as a streaming task.

  • StreamExport: Exports data from CPFS to the source storage as a streaming task.

  • Evict: Releases the data blocks of files in CPFS. After eviction, only the metadata of the files remains in the file system. You can still query the files, but their data blocks are cleared and no longer occupy storage capacity. When you access the file, CPFS reloads its data from the source storage on demand.

  • Inventory: Generates a file inventory that lists files managed by the data flow and their cache status.

Note

StreamImport and StreamExport are supported only by CPFS computing edition 2.6.0 and later.

Import
DataType

string

The type of data that the data flow task operates on. Valid values:

  • Metadata: The metadata of a file, such as its timestamp, ownership, and permissions. Only the file metadata is imported. You can see the file, but when you access the file data, CPFS loads the data from the source storage on demand.

  • Data: The data blocks of a file.

  • MetaAndData: The metadata and data blocks of a file.

Note

CPFS computing edition supports only the MetaAndData type.

MetaAndData

Progress

integer

The progress of the data flow task, based on the number of completed operations.

50
Status

string

The status of the data flow task. Valid values:

  • Pending: The task is created and waiting to be executed.

  • Executing: The task is in progress.

  • Failed: The task has failed. Review the data flow task report for details on the cause.

  • Completed: The task has finished. Review the data flow task report to verify the transfer.

  • Canceled: The task was canceled before it could be completed.

  • Canceling: The task is being canceled.

Failed
ReportPath deprecated

string

The path where the data flow task report is saved in the CPFS file system.

  • For CPFS, task reports are generated in the .dataflow_report directory of the file system.

  • For CPFS computing edition, this parameter returns an OSS download link for the task report.

/path_in_cpfs/reportfile.cvs
CreateTime

string

The time when the task was created.

2021-08-04 18:27:35
StartTime

string

The time when the task started.

2021-08-04 18:27:35
EndTime

string

The time when the task ended.

2021-08-04 18:27:35
FsPath

string

The intelligent directory path.

/aa/
ConflictPolicy

string

The conflict policy for files with the same name.

  • SKIP_THE_FILE: Skips the file with the same name.

  • KEEP_LATEST: Compares the update times and keeps the latest version.

  • OVERWRITE_EXISTING: Overwrites the existing file.

KEEP_LATEST
Directory

string

The directory processed by the data flow task.

/path_in_cpfs/
DstDirectory

string

The destination directory mapped to the data flow task.

/path_in_cpfs/

ErrorMsg

string

The cause of the task exception.

Note

A null or empty value in this field indicates that no error occurred.

{"ErrorKey":"PATH_NOT_ACCESSIBLE","ErrorDetail":"lstat /cpfs/370lx1ev9ss27o****/test/abcdfnotfound: no such file or directory"}
ProgressStats

object

The progress details of the data flow task.

FilesTotal

integer

The total number of files scanned at the source.

3
FilesDone

integer

The number of files that have been processed, including skipped files.

3
ActualFiles

integer

The number of files that were actually transferred.

3
BytesTotal

integer

The total amount of data scanned at the source, in bytes.

131092971520
BytesDone

integer

The amount of data that has been processed, including data from skipped files, in bytes.

131092971520
ActualBytes

integer

The amount of data that was actually transferred, in bytes.

131092971520
RemainTime

integer

The estimated time remaining, in seconds.

437
AverageSpeed

integer

The average transfer speed, in bytes/s.

342279299
Reports

object

Report

array<object>

A list of reports.

Note

  • Streaming tasks do not support reports.

  • When the WithReport parameter is true, this field contains report data for CPFS computing edition file systems.

  • The WithReport parameter is supported only by CPFS computing edition.

object

Name

string

The name of the report.

  • CPFS:

    TotalFilesReport: The task report.

  • CPFS computing edition:

    • FailedFilesReport: A report of files that failed to transfer.

    • SkippedFilesReport: A report of files that were skipped.

    • SuccessFilesReport: A report of files that transferred successfully.

TotalFilesReport
Path

string

The report URL.

https://a-hbr-temp-cn-hangzhou-staging.oss-cn-hangzhou.aliyuncs.com/temp/report/162319438359****/job-000bb6fwqficjbxk****/job-000bb6fwqficjbxk****_failed.zip?Expires=1721201422&OSSAccessKeyId=LTA****************&Signature=Fp%2BvauORTIVxooXY2tec6z0T%2Bp4%3D
Includes

string

The subdirectories to include in the task. The task processes only the contents within these directories.

Note

This feature is supported only by CPFS computing edition.

["/test/","/test1/"]

TransferFileListPath

string

Specifies an OSS directory that contains a CSV file. The task synchronizes only the files listed in this CSV file.

Note

This feature is supported only by CPFS computing edition.

/path_in_cpfs/

Examples

Success response

JSON format

{
  "RequestId": "2D69A58F-345C-4FDE-88E4-BF518948****",
  "NextToken": "TGlzdFJlc291cmNlU****mVzJjE1MTI2NjY4NzY5MTAzOTEmMiZORnI4NDhVeEtrUT0=",
  "TaskInfo": {
    "Task": [
      {
        "FilesystemId": "bmcpfs-290w65p03ok64ya****",
        "DataFlowId": "df-194433a5be31****",
        "TaskId": "task-38aa8e890f45****",
        "SourceStorage": "oss://178321033379****:bucket-01",
        "FileSystemPath": "/a/b/c/",
        "Originator": "User",
        "TaskAction": "Import",
        "DataType": "MetaAndData ",
        "Progress": 50,
        "Status": "Failed",
        "ReportPath": "/path_in_cpfs/reportfile.cvs",
        "CreateTime": "2021-08-04 18:27:35",
        "StartTime": "2021-08-04 18:27:35",
        "EndTime": "2021-08-04 18:27:35",
        "FsPath": "/aa/",
        "ConflictPolicy": "KEEP_LATEST",
        "Directory": "/path_in_cpfs/",
        "DstDirectory": "/path_in_cpfs/\n",
        "ErrorMsg": "{\"ErrorKey\":\"PATH_NOT_ACCESSIBLE\",\"ErrorDetail\":\"lstat /cpfs/370lx1ev9ss27o****/test/abcdfnotfound: no such file or directory\"}",
        "ProgressStats": {
          "FilesTotal": 3,
          "FilesDone": 3,
          "ActualFiles": 3,
          "BytesTotal": 131092971520,
          "BytesDone": 131092971520,
          "ActualBytes": 131092971520,
          "RemainTime": 437,
          "AverageSpeed": 342279299
        },
        "Reports": {
          "Report": [
            {
              "Name": "TotalFilesReport",
              "Path": "https://a-hbr-temp-cn-hangzhou-staging.oss-cn-hangzhou.aliyuncs.com/temp/report/162319438359****/job-000bb6fwqficjbxk****/job-000bb6fwqficjbxk****_failed.zip?Expires=1721201422&OSSAccessKeyId=LTA****************&Signature=Fp%2BvauORTIVxooXY2tec6z0T%2Bp4%3D"
            }
          ]
        },
        "Includes": "[\"/test/\",\"/test1/\"]\n",
        "TransferFileListPath": "/path_in_cpfs/\n"
      }
    ]
  }
}

Error codes

HTTP status code

Error code

Error message

Description
400 IllegalCharacters The parameter contains illegal characters. The parameter contains illegal characters.
400 MissingFileSystemId FileSystemId is mandatory for this action.
400 InvalidFilesystemVersion.NotSupport This Api does not support this fileSystem version. This Api does not support this fileSystem version.
403 OperationDenied.DataFlowNotSupported The operation is not supported.
404 InvalidParameter.InvalidNextToken The specified NextToken is invaild.
404 InvalidFileSystem.NotFound The specified file system does not exist. The specified file system does not exist.
404 InvalidDataFlow.NotFound The specified data flow does not exist.
404 InvalidParameter.InvalidMaxResults The specified MaxResults is invalid.
404 InvalidFilterParam The specified Filter.N.Key is invalid.

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.