Manage data flows for CPFS for Lingjun-Cloud Parallel File Storage(CPFS)-阿里云帮助中心

To transfer data between a CPFS for Lingjun file system and an Object Storage Service (OSS) bucket, you must first create a data flow. This topic describes how to create and manage data flows for CPFS for Lingjun in the NAS console.

Prerequisites

To prevent data conflicts when multiple data flows export data to the same OSS bucket, you must enable versioning for the bucket. For more information, see Introduction to versioning.

Limitations

CPFS for Lingjun 2.4.0 and later support dataflows within the same account. CPFS for Lingjun 2.6.0 and later support dataflows across different accounts.
A single CPFS for Lingjun file system supports a maximum of 10 dataflows.
A file path in a CPFS for Lingjun file system can be linked to only one OSS bucket.
Dataflows cannot be created between a CPFS for Lingjun file system and an OSS bucket in a different region.

For more information about the limits on the data flow feature, see Limitations.

Create a same-account data flow

Log on to the NAS console.
In the left-side navigation pane, choose File System > File System List.
In the top navigation bar, select a region.
On the File System List page, click the name of the file system.
On the details page of the file system, click Dataflow.
On the Dataflow tab, click Create Dataflow.

In the Create Dataflow dialog box, configure the following parameters.

Parameter	Description
CPFS File System Path	The path in the CPFS for Lingjun file system to associate with the OSS bucket. Constraints: 1 to 1,023 characters in length. Must start and end with a forward slash (/).
OSS Bucket	The source OSS bucket to associate with the path of the CPFS for Lingjun file system. Select Select a bucket in the current account and then choose the name of the target OSS bucket from the drop-down list.
OSS Object Prefix	The path within the OSS bucket to use as the data flow source. Constraints: 1 to 1,023 characters in length. Must start and end with a forward slash (/). The prefix must already exist in the OSS bucket.
OSS Bucket SSL	Specifies whether to use HTTPS to access OSS.
OSS Bucket Tag	You must add the `cpfs-dataflow: true` tag to the OSS bucket before you can create a data flow. Do not delete or modify this tag while the data flow is in use. Otherwise, the data flow for the CPFS for Lingjun file system cannot access data in the bucket. How to add a tag Go to the target bucket. In the left-side navigation pane, choose Bucket Settings > Bucket Tagging. Click Create Tag. Enter `cpfs-dataflow` for the tag key and `true` for the tag value. Click Save to add the tag, and then return to the NAS console to continue creating the data flow.
SLR Authorization	When you Create Dataflow for the first time, you must authorize the service-linked role that allows CPFS to access resources in Object Storage Service (OSS). For more information, see File Storage CPFS service-linked role.

Click OK.
After you click OK, the system verifies your configuration. This process takes 1–2 minutes. The dialog box closes automatically after the verification is complete. Do not close it manually.

Create a cross-account data flow

If you need to move data from an OSS bucket in Account B to a CPFS for Lingjun file system in Account A, you must first log on to Account B to authorize AliyunNasCrossAccountDataFlowDefaultRole and add the UID of Account A to the role's trust policy. Then, you can log on to Account A to create the cross-account data flow and data import or export tasks.

This section uses a data flow between a CPFS for Lingjun file system under Alibaba Cloud account A and an OSS bucket under Account B as an example.

Procedure

Grant permissions from Account B (the OSS bucket owner).
1. Log on to the NAS console using Account B.
2. On the Overview page, in the References section, click Authorization Management.
3. In the Authorization Management panel, in the Cross-account Authorization on Data Flows section, click Authorize Now.
4. Click Authorize.
5. Return to the Authorization Management panel in the NAS console. In the Cross-account Authorization on Data Flows section, click View Details to go to the details page of AliyunNasCrossAccountDataFlowDefaultRole.
6. On the Trust Policy tab, click Edit Trust Policy.
7. Modify the Service field to <UID of Alibaba Cloud account A>@nas.aliyuncs.com.
  For example, if the UID of Alibaba Cloud account A is 178321033379****, change the value of Service from nas.aliyuncs.com to 178321033379****@nas.aliyuncs.com. This indicates that the data flow service of Alibaba Cloud account A (178321033379****@nas.aliyuncs.com) can assume this role.
```
{
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "178321033379****@nas.aliyuncs.com" 
        ]
      }
    }
  ],
  "Version": "1"
}
```

Create the cross-account data flow using Account A.

Log on to the NAS console using Account A.
In the left-side navigation pane, choose File System > File System List.
In the top navigation bar, select a region.
On the File System List page, click the name of the file system.
On the details page of the file system, click Dataflow.
On the Dataflow tab, click Create Dataflow.

In the Create Dataflow dialog box, configure the following parameters.

Parameter	Description
CPFS File System Path	The path in the CPFS for Lingjun file system to associate with the OSS bucket. Constraints: 1 to 1,023 characters in length. Must start and end with a forward slash (/).
OSS Bucket	The source OSS bucket to associate with the path of the CPFS for Lingjun file system. Select Specify a bucket in another account. Enter the UID of the account that owns the source OSS bucket in the Account ID field, and enter the name of the source OSS bucket in the Bucket Name field.
OSS Object Prefix	The path within the OSS bucket to use as the data flow source. Constraints: 1 to 1,023 characters in length. Must start and end with a forward slash (/). The prefix must already exist in the OSS bucket.
OSS Bucket SSL	Specifies whether to use HTTPS to access OSS.

Click OK.
After you click OK, the system verifies your configuration. This process takes 1–2 minutes. The dialog box closes automatically after the verification is complete. Do not close it manually.

Related operations

After creating a data flow, you can view its configuration, update its description, and delete it in the console.

Actions	Description	Steps
View a data flow	You can view created data flows and create data flow tasks for a specific data flow.	On the Dataflow tab, you can view the configuration of a specific data flow.
Modify a data flow	You can modify only the description of a data flow.	On the Dataflow tab, find the target data flow. Click Modify and update the description. Click OK.
Delete a data flow	Deleting a data flow deletes all associated tasks and stops data synchronization. Important You cannot delete a data flow if it has running streaming jobs or batch jobs.	On the Dataflow tab, find the target data flow. Click Delete. In the confirmation message, click OK.

What to do next

After you create a data flow, create export or import tasks as needed to transfer data between the CPFS for Lingjun file system and the OSS bucket. For more information, see Create a task.