Configure the SelectDB Input Component

Prerequisites

• You have created a SelectDB data source. For more information, see Create a SelectDB Data Source.

• The account used to configure the SelectDB input component must have sync-read permission on the data source. If the account does not have this permission, request it. For more information, see Request Data Source Permissions.

Procedure

1. In the top menu bar on the Dataphin homepage, choose Develop > Data Integration.

2. In the top menu bar of the Integration page, select a Project. In Dev-Prod mode, also select an environment.

3. In the navigation pane on the left, click Batch Pipeline. In the Batch Pipeline list, click the offline pipeline that you want to develop. The configuration page for the pipeline opens.

4. In the upper-right corner of the page, click Component Library to open the Component Library panel.

5. In the navigation pane on the left of the Component Library panel, click Input. In the input component list on the right, find the SelectDB component and drag it onto the canvas.

6. Click the icon in the SelectDB input component card to open the SelectDB Input Configuration dialog box.

7. In the SelectDB Input Configuration dialog box, configure the parameters.

   Parameter	Description
   Step Name	The name of the SelectDB input component. Dataphin generates a default name that you can change. The name must follow these rules: Use only Chinese characters, letters, underscores (_), and digits. Use no more than 64 characters.
   Datasource	The drop-down list shows all SelectDB data sources in the current Dataphin instance and indicates whether you have sync-read permission for each. If you do not have sync-read permission for a data source, click Request next to the data source to request the permission. For more information, see Request Data Source Permissions. If you do not have a SelectDB data source, click Create Data Source to create one. For more information, see Create a SelectDB Data Source.
   Source Table Count	The number of source tables to sync. Valid values: Single Table: Syncs data from one source table to one destination table. Multiple Tables: Syncs data from multiple source tables to one destination table using the union algorithm. For more information about union, see INTERSECT, UNION, and EXCEPT.
   Table Matching Method	Choose Generic Rule or Database Regex. Note This parameter is available only when you select Multiple Tables for Source Table Count.
   Table	Select the source table: If you selected Single Table for Source Table Count, search by entering a keyword in the table name field. Or enter the exact table name and click Exact Match. After you select a table, the system automatically checks its status. Click the icon to copy the name of the selected table. If you selected Multiple Tables for Source Table Count, enter an expression based on the table matching method: If you chose Generic Rule: Enter an expression in the field to filter tables with the same structure. The system supports enumeration, regex-like syntax, and mixed formats. For example: table_[001-100];table_102;. If you chose Database Regex: Enter a regex supported by the database. The system matches tables in the destination database using this regex. At runtime, the task matches new tables dynamically based on the regex. After entering the expression, click Exact Match. In the Confirm Match Details dialog box, view the list of matched tables.
   Split Key (Optional)	The number of records to read at a time. Configure a batch size—such as 1,024 records—to read data from the source database in batches instead of one record at a time. This reduces interactions with the data source, improves I/O efficiency, and lowers network latency.
   Batch Read Size (Optional)	The number of records to read per batch. For example, set this to 1,024 to read data in batches instead of one record at a time, reducing data source interactions and network latency.
   Input Filter (Optional)	Filters the data to extract from the source table. You can configure the filter in the following ways: Set a static value to extract matching data. For example: ds=20210101. Set a variable parameter to extract part of the data. For example: ds=${bizdate}.
   Output Fields	Lists all fields from the selected table and filters. You can manage the field list as follows: Manage Fields: Remove fields that you do not need to pass to downstream components: Delete One Field: Click the icon in the Actions column to delete a single field. Delete Multiple Fields: Click Field Management, select the fields to remove, and click the left arrow icon to move them to the unselected list. Click OK. Batch Add: Click Batch Add to add fields in JSON, TEXT, or DDL format. Note After you click OK, the batch-added fields overwrite any existing field configurations. JSON format example: // Example: [{ "index": 1, "name": "id", "type": "int(10)", "mapType": "Long", "comment": "comment1" }, { "index": 2, "name": "user_name", "type": "varchar(255)", "mapType": "String", "comment": "comment2" }] Note The index field indicates the column number of the specified object, the name field indicates the field name after import, and the type field indicates the field type after import. For example, "index":3,"name":"user_id","type":"String" imports the fourth column in the file, sets the field name to user_id, and sets the field type to String. TEXT format example: // Example: 1,id,int(10),Long,comment1 2,user_name,varchar(255),Long,comment2 The row delimiter separates each field’s information. The default is a line feed (\n). You can also use a semicolon (;) or a period (.). The column delimiter separates field names and types. The default is a comma (,). You can also use ','. The field type is optional and defaults to ','. DDL format example: CREATE TABLE tablename ( user_id serial, username VARCHAR(50), password VARCHAR(50), email VARCHAR (255), created_on TIMESTAMP, ); Add a New Output Field: Click +Add Output Field. Enter values for Column, Type, and Comment. Select a Mapping Type. Click the icon to save the row.

8. Click OK.