Configure the OpenSearch input component

Prerequisites

• You have created an OpenSearch data source. For more information, see Create an OpenSearch data source.

• The account used to configure the OpenSearch input component properties must have read-through permission for the data source. If the account does not have the permission, request it. For more information, see Request permissions for a data source.

Procedure

1. In the top menu bar of the Dataphin home page, choose R&D > Data Integration.

2. In the top menu bar of the integration page, select a Project. If you are in Dev-Prod mode, also select an environment.

3. In the left navigation pane, click Offline Integration. In the Offline Integration list, click the offline pipeline that you want to develop to open its configuration page.

4. In the upper-right corner of the page, click Component Library to open the Component Library panel.

5. In the navigation pane on the left of the Component Library panel, choose Input. Find the OpenSearch component in the list of input components on the right and drag it to the canvas.

6. Click the icon on the OpenSearch input component card to open the OpenSearch Input Configuration dialog box.

7. In the OpenSearch Input Configuration dialog box, configure the parameters.

   Parameter		Description
   Basic Configuration	Step Name	The name of the OpenSearch input component. Dataphin automatically generates a step name, which you can change as needed. The naming conventions are as follows: Can contain only Chinese characters, letters, underscores (_), and digits. Cannot exceed 64 characters.
   	Datasource	The drop-down list displays all OpenSearch data sources in the current Dataphin project, including those for which you do not have read-through permission. Click the icon to copy the data source name. For a data source without read-through permission, click Request next to the data source to request permission. For more information, see Request permissions for a data source. If no OpenSearch data source exists, click Create Data Source in the drop-down list to create one. For more information, see Create an OpenSearch data source.
   	Index document	The name of the index in OpenSearch. Click the icon to copy the name of the selected index document.
   	Retrieve query condition	The query parameter for OpenSearch, used for full or incremental queries. For example, { "match_all": {}} indicates a full query.
   	Cursor Time	The duration to keep the scroll open. This is the paging parameter for OpenSearch. If this value is too small, the scroll may expire between page fetches, which causes data loss. If this value is too large and too many queries are initiated at the same time, the number of open scrolls may exceed the server-side max_open_scroll_context setting. This causes query errors. For example, 5m specifies a scroll time of 5 minutes. Units: days (d), hours (h), minutes (m), seconds (s), milliseconds (ms), microseconds (micros), and nanoseconds (nanos).
   Advanced Configuration	Batch read size	The number of data entries to read at a time. Default: 1024. Setting a larger batch size reduces the number of interactions with the data source, improves I/O efficiency, and lowers network latency.
   	Connection timeout	The client connection timeout. Default: 60000 ms.
   	Read timeout	The client read timeout. Default: 60000 ms.
   Output fields		Displays all fields from the selected table that match the filter criteria. The following operations are supported: Get field information. Click Get field information to retrieve the field information for the selected index. Batch Add fields. Click Batch Add. Configure fields in JSON format. Example: // Example: [{ "name": "col_integer", "type": "integer" }, { "name": "col_long", "type": "long" }, { "name": "col_double", "type": "double" }] Note name is the name of the imported field, and type is the field type after import. For example, "name":"user_id","type":"String" imports the field named user_id and sets its type to String. Configure fields in TEXT format. Example: // Example: col_integer,integer col_long,long col_double,double The row delimiter separates the information for each field. The default is a line feed (\n). Semicolons (;) and periods (.) are also supported. The column delimiter separates the field name from the field type. The default is a comma (,). Click OK. Create Output Field. Click Create Output Field, and then enter a Column and select a Type as prompted. Manage output fields. The following operations are available for added fields: Click and drag the move icon next to the Field to change its position. Click the edit icon in the Actions column to edit an existing field. Click the delete icon in the Actions column to delete an existing field.

8. Click Confirm to complete the configuration of the OpenSearch input component.