Configure an Elasticsearch writer

Limitations

DataWorks supports only Alibaba Cloud Elasticsearch 5.x, 6.x, and 7.x data sources. Self-managed Elasticsearch data sources are not supported.

Procedure

1. Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.

2. Hover over the icon and choose Create Node > Data Integration > Real-time Synchronization.

   Alternatively, expand a scheduled workflow, right-click it, and choose Create Node > Data Integration > Real-time Synchronization.

3. In the Create Node dialog box, set Synchronization Method to Single Table (Topic) to Single Table (Topic) ETL, enter a Name, and select a Path.

4. Click Confirm.

5. On the configuration tab of the real-time synchronization node, drag Output to the canvas and connect it to the configured reader or transformation node.

6. Click the Elasticsearch node. In the Node Configuration dialog box that appears, configure the parameters.

   Parameter	Description
   Data Source	Select a configured Elasticsearch data source. If you have not configured a data source, click New data source on the right to go to the Management Center > Data Source page and create one. For more information, see Configure an Elasticsearch data source.
   Index	Select the destination index. To create an index, click Create Index on the right. You can then use the default index information or customize the index name, index type, dynamic mapping status, number of shards, number of replicas, and the index creation statement before you create the index. Index Type: This parameter is unavailable for Elasticsearch 7.x and later but is supported and customizable for 5.x, 6.x, and earlier versions. Dynamic Mapping Status: Specifies the value of the dynamic parameter for the root of the Elasticsearch index. This parameter controls whether new fields can be dynamically added and if Elasticsearch automatically infers their types. For versions earlier than 7.10, the possible values are: true, false, and strict. For versions 7.10 and later, the valid values include: true, false, strict, and runtime. true indicates that it can be stored and indexed. false indicates that the data can be stored but cannot be indexed. strict: Throws an exception and does not store the data when a new field appears. runtime specifies that the new field is a runtime field, which is not indexed. For more information, see dynamic. Shards: Specifies the number of primary shards. To enable distributed search, Elasticsearch splits a complete index into multiple shards and distributes them across different nodes. The number of shards must be specified when the index is created and cannot be changed later. For more information, see Key concepts. Number of Replicas: Specifies the number of replicas for each shard. Replicas provide fault tolerance and handle read request loads. The default value is 1. This is recommended if you have insufficient capacity, do not need multiple data backups, or experience poor write performance. Statement Used to Create Index: In properties, configure fields and customize their data types.
   Enable Elasticsearch partitioning	Specifies whether to enable the routing mechanism in Elasticsearch. The routing parameter, which defaults to the document's _id, can be set to a custom value. The routing mechanism uses a hash function to generate a number from the routing value. It then calculates the remainder of this number divided by the number of primary shards. This calculation determines the shard where the document is stored.
   Primary key settings	Specifies how to generate values for the _id field in the Elasticsearch index during synchronization. Primary Key: Use a single column from the source table as the primary key. Composite Primary Key: Combine multiple columns from the source table to form a composite primary key. Note If you select a primary key option but do not map the corresponding source fields (for example, you select Primary Key but no primary key exists in the source, or you select Composite Primary Key but do not configure the key columns), the setting is ignored. Instead, random values are generated for _id, which can lead to data duplication.
   Field Mapping	Configures the field mapping between the source and destination. The synchronization task transfers data based on this mapping.

7. Click the icon in the toolbar.