Configure the dataset input component

Prerequisites

• The dataset input component is available only after the unstructured data feature has been purchased for the current instance.

• A dataset has been created. For more information, see Dataset.

Procedure

1. In the top menu bar of the Dataphin homepage, choose Develop > Data Integration.

2. In the top menu bar of the Integration page, select a project. In Dev-Prod mode, you must also select an environment.

3. In the left navigation pane, click Offline Integration. In the Offline Integration list, click the target offline pipeline to open its configuration page.

4. In the upper-right corner of the page, click Component Library to open the Component Library panel.

5. In the navigation pane on the left of the Component Library panel, click Input. From the list of input components on the right, find the Dataset component and drag it to the canvas.

6. Click the icon on the dataset input component to open the Dataset Input Configuration dialog box.

7. In the Dataset Input Configuration dialog box, configure the parameters.

   Parameter	Description
   Step Name	The name of the dataset input component. Dataphin automatically generates a step name. You can change the name as needed. The naming conventions are as follows: Can only contain Chinese characters, letters, underscores (_), and numbers. The name can be up to 64 characters long.
   Dataset	Select a table or hybrid dataset from the current project. You can search for a dataset by entering a keyword from its name. After you select a dataset, select a Version. Click the icon to copy the dataset name.
   Batch read size (Optional)	The number of records to read in a single batch. Reading data in batches, such as 1024 records at a time, reduces interactions with the data source. This improves I/O efficiency and lowers network latency.
   Input filter	Enter a conditional expression supported by PostgreSQL to filter data. Write only the content that follows the WHERE keyword. Do not include the WHERE keyword itself. System global variables, such as the data timestamp ${bizdate}, are also supported.
   Output fields	The Output Fields area displays all fields from the metadata of the selected dataset.

8. Click Confirm to complete the configuration of the dataset input component.