Create a data collection task-Dataphin(Dataphin)-阿里云帮助中心

Unstructured data collection centralizes unstructured files from various locations, such as OSS or a data lake. As the first step in processing unstructured data, it simplifies subsequent processing. This topic describes how to create a data collection node.

Limits

The dataset input and output components are available only if you have purchased the unstructured data feature for the current instance.
You can create dataset input and dataset output components only in Basic projects.

Permissions

The following project roles can use the dataset input and output components to access all datasets in a project:

Dev and Basic projects: project administrator, developer, and analyst.
Prod projects: project administrator and O&M.
Custom project roles with the Dataset - Use permission.

Procedure

Step 1: Create a data collection node

On the Dataphin home page, choose Developer > Data Integration in the top menu bar.
In the top menu bar of the integration page, select a Project. If you are in Dev-Prod mode, also select an Environment.
In the navigation pane on the left, choose Unstructured Data > Data Collection. In the data collection list on the right, click the icon to open the Create Data Collection Node dialog box.

In the Create Data Collection Node dialog box, configure the following parameters.

Parameter	Description
Task Name	The name can be up to 64 characters long. It cannot contain special characters, such as the vertical bar (\|), forward slash (/), backslash (\), colon (:), question mark (?), angle brackets (<>), asterisk (*), or single quotation mark (').
Schedule Type	Select the schedule type for the pipeline. The Schedule Type can be: Recurring task node: A node that runs on a regular schedule. One-time task: A node with no dependencies that you must trigger manually.
Description	Enter a brief description of the data collection node. The description can be up to 1,000 characters long.
Select Directory	Select the folder that contains the task. If no folder exists, you can create a new folder as follows: Above the list of compute nodes on the left, click the icon to open the Create Folder dialog box. In the Create Folder dialog box, enter a folder Name and select a Directory location as needed. Click OK.

Click OK.

Step 2: Configure inputs and outputs

From the Component Library on the left of the canvas, drag the input and output components to the canvas. Then, configure the components in the panel on the right.

Input components

File input

Parameter	Description
Step Name	The default value is file_input_1. You can change the name. The name can contain Chinese characters, English letters, underscores (_), and digits.
Data Source Type	You can select OSS, Amazon S3, FTP, or Hadoop Distributed File System (HDFS).
Datasource	Select a data source for which you have read-through permissions. If no suitable data source is available, click New Data Source to create one. For more information, see Create an OSS data source, Create an Amazon S3 data source, Create an FTP data source, and Create an HDFS data source. Click the icon to copy the data source name.
Object	You can use wildcard characters and specify multiple objects. For example, `category/` reads all objects that start with `category/`. Note* This parameter is available only when the data source type is OSS or Amazon S3.
File Path	You can enter multiple file paths separated by semicolons (;). You can use wildcard characters in the paths. For example, `/bazhen/` reads all files in the `bazhen` directory. Note* This parameter is available only when the data source type is FTP or HDFS.
Output Field	The output fields are collected from the metadata of the files to be collected. You can search for fields by keyword.

Local file input

Note

Only one-time tasks support local file input.

Parameter	Description
Step Name	The default value is local_file_input_1. You can change the name. The name can contain Chinese characters, English letters, underscores (_), and digits.
File Path	Click Select File or drag the object files to the file path area. You can select multiple files. Each file cannot exceed 500 MB in size.
Output Field	The output fields are parsed from the metadata of the uploaded files. You can search for fields by keyword.

Dataset input

Parameter	Description
Step Name	The default value is dataset_input_1. You can change the name. The name can contain Chinese characters, English letters, underscores (_), and digits.
Dataset	Select a table dataset or mixed dataset from the current project. You can search for datasets by keyword. After selecting a dataset, you must also select its version. Click the icon to copy the current dataset name.
Object	You can use wildcard characters and specify multiple objects. For example, `category/*` reads all objects that start with `category/`.
Output Field	The output fields are parsed from the metadata of the files in the specified path. You can search for fields by keyword.

Output components

File output

Parameter	Description
Step Name	The default value is file_output_1. You can change the name. The name can contain Chinese characters, English letters, underscores (_), and digits.
Data Source Type	You can select OSS, Amazon S3, FTP, or HDFS.
Datasource	Select a data source for which you have synchronous write permissions. If no suitable data source is available, click New Data Source to create one. For more information, see Create an OSS data source, Create an Amazon S3 data source, Create an FTP data source, and Create an HDFS data source. Click the icon to copy the data source name.
Object Prefix	Enter only the directory path. The object file name is the same as the source file name by default. Note This parameter is available only when the data source type is OSS or Amazon S3.
File Path	Enter only a single file path. The object file name is the same as the source file name by default. Note This parameter is available only when the data source type is FTP or HDFS.
Loading Policy	OSS, Amazon S3, and FTP data sources support overwriting data, appending data, and reporting an error for file name conflicts. HDFS data sources support overwriting data and appending data. Overwrite data: Before writing data, clear files with conflicting names in the directory, and then add the new data files. Append data: Directly add new data files to the destination directory. A random universally unique identifier (UUID) suffix is used to ensure that file names do not conflict. Report error for file name conflict: If a file with a conflicting name exists in the directory, an error is reported at runtime.

Dataset output

Parameter	Description
Step Name	The default value is data_integration_output_1. You can change the name. The name can contain Chinese characters, English letters, underscores (_), and digits.
Dataset	Select a table dataset or mixed dataset from the current project. You can search for datasets by keyword. After selecting a dataset, you must also select its version. Click the icon to copy the current dataset name.
Object Prefix (optional)	Enter only the directory path. The object file name is the same as the source file name by default.
Loading Policy	Overwrite data: Before writing data, clear files with conflicting names in the directory, and then add the new data files. Append data: Directly add new data files to the destination directory. A random UUID suffix is used to ensure that file names do not conflict. Report error for file name conflict: If a file with a conflicting name exists in the directory, an error is reported at runtime.
Sync Metadata	Only hybrid datasets support metadata sync, and if you select Yes, you must also configure the File Path Field and Field Mapping.
File Path Field	The field that points to the file path after the file is written to the dataset. You can select a field for which Is URL is selected in the chosen dataset.
Field Mapping	You can manually map metadata fields based on the output fields from the upstream component. Source field: The list of output fields from the upstream component. Metadata field: You can select a dataset field (excluding the selected file path field). Fields cannot be selected more than once. Click Configure Field Mapping to perform batch operations in the Field Mapping dialog box. Both map by row and map by name are supported. Map by name: Maps fields that have the same name. Map by row: Maps fields in the same row when the field names in the source and destination tables do not match.

Step 3: Configure the data collection node properties

In the menu bar on the canvas, click the Scan Configuration button to open the Scan Configuration tab in the Property panel.
In the Scheduling Configuration panel, you can configure the Basic Information, Runtime Parameters, Scheduling Configuration, Scheduling Dependencies, Runtime Configuration, and Resource Configuration for the data collection task. These configurations are described as follows:
- Basic Information: Specifies the basic information of the data collection node. The configuration is the same as for an offline integration node. You can configure the developer, owner, tags, and description. For more information, see Configure basic information for an offline integration pipeline.
- Runtime Parameters: If your node uses parameter variables, you can assign values to them in this section. The variables are automatically replaced with their assigned values during node scheduling. The configuration is the same as for an offline integration node. For more information, see Configure runtime parameters for an offline pipeline.
- Scheduling Configuration: For a recurring task node, this section defines how the data collection node is scheduled in the production environment. The configuration is the same as for an offline integration node. For more information, see Offline integration pipeline scheduling configuration.
- Scheduling Dependencies: For a recurring task node, this section defines the upstream dependencies of the data collection node in a scheduled task. The configuration is the same as for an offline integration node. For more information, see Configure scheduling dependencies for an offline integration pipeline.
- Runtime Configuration: You can configure a node-level timeout period and a retry policy for a failed node. This prevents long-running nodes from wasting resources and improves node reliability. The configuration is the same as for an offline integration node. For more information, see Configure runtime settings for an offline integration pipeline.
- Resource Configuration: You can configure the resource group for the current data collection node. The node uses resources from this group for scheduling. The configuration is the same as for an offline integration node. For more information, see Configure resources for an offline integration pipeline node.
Click the Channel Configuration tab on the Properties panel to define the runtime fault tolerance, concurrency, JVM resources, and database configuration for the current data collection task. The channel configuration is the same as for offline integration tasks, except that traffic monitoring is not supported. For more information, see Offline Integration Pipeline Channel Configuration.

Step 4: Save and submit the data collection node

Click the Save icon above the canvas to save the data collection node.
Click the Submit icon above the canvas. In the Commit Note dialog box, enter your notes and click OK and Submit.
When you submit the node, Dataphin parses its data lineage and performs a validation check. For more information, see Integration node submission instructions.

What to do next

After you create a data collection node, you can go to the Data Collection Node List to view and manage your data collection nodes. This node list is the same as the offline integration node list, with a few exceptions for filter options. The filters for source table, topic, or index and destination table, topic, or index are not supported. The `source data source/project/data category` filter is changed to `source data source/dataset`, and the `destination data source/project/data category` filter is changed to `destination data source/dataset`. For more information, see View the offline integration node list.
After a data collection node runs, you can go to Data Collection Run Records to view information about the node's execution. The information in the run list is the same as for an offline integration node. For more information, see View and manage the run record list.
If you use the Dev-Prod development mode, you must publish the node. For more information, see Manage release tasks.
If you use the Basic development mode, a node that is successfully submitted can be scheduled in the production environment. You can go to the Operation Center to view your published nodes. For more information, see Operation Center.