Configure the OpenSearch output component

Prerequisites

• An OpenSearch data source has been created. For more information, see .

• The account used to configure the OpenSearch output component must have sync write permission for the data source. If the account does not have the required permission, you must request it. For more information, see Request data source permissions.

Procedure

1. On the Dataphin home page, choose Developer > Data Integration from the top menu bar.

2. On the Data Integration page, select a Project from the top menu bar. If you are in Dev-Prod mode, you must also select an environment.

3. In the navigation pane on the left, click Batch Pipeline. In the Batch Pipeline list, click the offline pipeline that you want to develop to open its configuration page.

4. In the upper-right corner of the page, click Component Library to open the Component Library panel.

5. In the navigation pane on the left of the Component Library panel, select Outputs. Find the OpenSearch component in the list of output components and drag it to the canvas.

6. Click and drag the icon of the target input component to connect it to the OpenSearch output component.

7. Click the icon on the OpenSearch output component card to open the OpenSearch Output Configuration dialog box.

   

8. In the OpenSearch Output Configuration dialog box, configure the parameters.

   Parameter	Description
   Step Name	The name of the OpenSearch output component. Dataphin automatically generates a step name. You can also change it as needed. The naming convention is as follows: Can contain only Chinese characters, uppercase and lowercase letters, underscores (_), and digits. Cannot exceed 64 characters in length.
   Datasource	The data source drop-down list displays all OpenSearch data sources. This includes data sources for which you have the sync write permission and those for which you do not. For a data source where you lack sync write permission, click Request next to the data source to request the permission. For more information, see Request, renew, and revoke data source permissions. If you do not have an OpenSearch data source, click Create to create one. For more information, see .
   Loading Policy	The policy for writing data to the target table. Loading Policy includes: Append Data: Appends data directly to the table. Update Data: Requires an ID to be specified in the output fields. This field is used as the primary key to overwrite and update data.
   Index Document	Select the index document. This is the index name in OpenSearch.
   Field Separator (Optional)	Enter the separator for fields in the file content based on your storage format. If you leave this blank, a comma (,) is used by default.
   Input Fields	Displays the input fields based on the output from the upstream component.
   Output Fields	Displays the output fields. Get Field Information: Retrieves the field information of the selected Index Document. Add Fields In Batches. Click Add In Batches. Configure in JSON format. Example: [{"name":"col_integer","type":"integer"}, {"name":"col_long","type":"long"}, {"name":"col_double","type":"double"}] Note name specifies the name of the field to import, and type specifies the data type of the field. For example, "name":"user_id","type":"String" imports the field named user_id and sets its data type to String. Configure in TEXT format. Example: col_long,long col_double,double The row delimiter separates the information of each field. The default is a line feed (\n). Semicolons (;) and periods (.) are also supported. The column delimiter separates the field name and field type. The default is a comma (,). Click OK. Create An Output Field. Click Create Output Field and enter a Column and select a Type as prompted. Copy Upstream Fields. Uses the upstream input fields as the output fields. Manage output fields. You can perform the following operations on the added fields: Next to Column, click and drag the Move icon to rearrange the position of the field. Click the Edit icon in the Actions column to edit an existing field. Click the Delete icon in the Actions column to delete the existing field.
   Mapping	The mapping relationship connects the input fields of the source table with the output fields of the target table for data synchronization. Mapping relationships include mapping by name and mapping by row. The scenarios are described as follows: Map By Name: Maps fields that have the same name. Map By Row: Maps fields in the same row when the field names in the source and target tables do not match, but the data in the corresponding rows needs to be mapped. Only fields in the same row are mapped.

9. Click Confirm to complete the property configuration for the OpenSearch output component.