Offline data writing from DataWorks to OpenSearch-DataWorks(DataWorks)-阿里云帮助中心

DataWorks Data Integration supports writing data to OpenSearch with OpenSearch Writer. This topic describes how to write offline data from DataWorks to OpenSearch.

Supported versions

For V2, see Request structure.
V3 uses a second-party package with the Maven dependency com.aliyun.opensearch:aliyun-sdk-opensearch:2.1.3.
To use the OpenSearch Writer plug-in, you must use JDK 1.6-32 or later. Run java -version to check the Java version.
Supported commercial editions of Alibaba Cloud OpenSearch: Industry Algorithm Edition, LLM Intelligent Q&A Edition, High-performance Search Edition, Vector Search Edition, and Retrieval Engine Edition.

Limits

OpenSearch Writer supports Serverless resource groups (recommended) and exclusive resource groups for Data Integration, but does not support custom resource groups.
OpenSearch columns are unordered. Therefore, when writing data, OpenSearch Writer must follow the specified column order. If you specify fewer columns than exist in OpenSearch, the remaining columns use their default values or null.
For example, suppose the fields to import are b and c, while the OpenSearch table has columns a, b, and c. You can configure the columns as "column":["c","b"]. The first and second Reader columns are then imported into fields c and b, and field a in the newly inserted record is set to its default value or null.
Currently, offline data can be written to OpenSearch only in script mode.

Supported field types

OpenSearch Writer supports most OpenSearch types. Check your data types. The following table lists the OpenSearch type mappings.

Type category	OpenSearch data type
Integer	INT
Floating point	DOUBLE and FLOAT
String	TEXT, LITERAL, and SHORT_TEXT
Date and time	INT
Boolean	LITERAL

Develop a data synchronization task

For information about the entry point for and the procedure of configuring a synchronization task, see the following configuration guides.

For the procedure, see Script mode configuration.
For all script-mode parameters and a sample script, see Appendix: Sample scripts and parameter descriptions.

FAQ

Handling incorrect column configurations

To ensure reliable writes and prevent data loss caused by excess columns, OpenSearch Writer reports an error if you write too many columns. For example, if an OpenSearch table has fields a, b, and c, OpenSearch Writer reports an error when it receives more than three fields.

Table configuration considerations

OpenSearch Writer can write to only one table at a time.

Rerunning tasks and failover

After a rerun, records with the same ID are automatically overwritten. Therefore, the columns inserted into OpenSearch must include an ID that uniquely identifies each record. Records with the same identifier overwrite each other.

Appendix: Sample scripts and parameter descriptions

Configure a batch synchronization task by using the code editor

If you want to configure a batch synchronization task by using the code editor, you must configure the related parameters in the script based on the unified script format requirements. For more information, see Script mode configuration. The following information describes the parameters that you must configure for data sources when you configure a batch synchronization task by using the code editor.

Sample Writer script (Industry Algorithm Edition, LLM Intelligent Q&A Edition, and High-performance Search Edition)

{
    "type": "job",
    "version": "1.0",
    "configuration": {
        "reader": {},
        "writer": {
            "plugin": "opensearch",
            "parameter": {
                "accessId": "*********",
                "accessKey": "********",
                "host": "http://yyyy.aliyuncs.com",
                "endpoint":"http://yyyy.aliyuncs.com",
                "indexName": "datax_xxx",
                "table": "datax_yyy",
                "column": [
                "appkey",
                "id",
                "title",
                "gmt_create",
                "pic_default"
                ],
                "batchSize": 500,
                "writeMode": add,
                "version":"v2",
                "ignoreWriteError": false
            }
        }
    }
}

Writer script parameters (Industry Algorithm Edition, LLM Intelligent Q&A Edition, and High-performance Search Edition)

Parameter	Description	Required	Default value
accessId	AccessKey ID of the access key pair.	Yes	None
accessKey	AccessKey Secret of the access key pair, equivalent to a logon password.	Yes	None
host	OpenSearch traffic endpoint. Find it on the details page of the instance in the OpenSearch console.	Yes	None
endpoint	OpenSearch management endpoint. Find it in the documentation for the relevant OpenSearch edition. For example, for Industry Algorithm Edition: Service endpoints.	Yes	None
indexName	Name of the OpenSearch application.	Yes	None
table	Name of the destination table. You cannot specify multiple tables because DataX does not support importing into multiple tables at the same time.	Yes	None
column	List of fields to import. To import all fields, configure `"column":["*"]`. To import only some OpenSearch columns, specify them, for example: `"column":["id","name"]`. OpenSearch supports column filtering and reordering. For example, if a table has fields a, b, and c, and you need only c and b, configure `["c","b"]`. During import, field a is populated with null.	Yes	None
batchSize	Number of records written per batch. OpenSearch supports batch writes and generally excels at queries rather than high write throughput. Set this based on the resources allocated to your account. Typically, a single OpenSearch record is smaller than 1 MB and a batch write is smaller than 2 MB.	This option is required for partitioned tables and must be omitted for non-partitioned tables.	300
writeMode	OpenSearch Writer uses "writeMode":"add/update" to ensure idempotent writes: "add": If a write fails and the task is rerun, OpenSearch Writer removes the existing record and imports the new record as an atomic operation. "update": Inserts the record by updating it as an atomic operation. Note OpenSearch batch inserts are not atomic: some records may succeed while others fail. Select writeMode carefully. V3 currently does not support update operations.	Yes	None
ignoreWriteError	Ignore write errors. Example configuration:`"ignoreWriteError":true`. Determines whether to ignore a failed batch write. If enabled, the task continues with other writes. Otherwise, the task stops and returns an error. The default is recommended.	No	false
version	OpenSearch version, for example `"version":"v3"`. V3 is recommended because V2 has more restrictions on push operations.	No	v2

Sample Writer script (Vector Search Edition and Retrieval Engine Edition)

{
  "stepType": "opensearch",
  "parameter": {
    "indexName": "",
    "column": [
      {
        "name": "col3double",
        "type": "DOUBLE"
      },
      {
        "name": "col2vector",
        "type": "MULTI_FLOAT"
      }
    ],
    "datasource": "zm_test_vector_01",
    "batchSize": "500",
    "table": "demotable"
  },
  "name": "Writer",
  "category": "writer"
}

Writer script parameters (Vector Search Edition and Retrieval Engine Edition)

Parameter	Description	Required	Default value
table	Name of the destination table. You cannot specify multiple tables because DataX does not support importing into multiple tables at the same time.	Yes	None
column	List of fields to import. To import all fields, configure `"column":["*"]`. To import only some OpenSearch columns, specify them, for example: `"column":["id","name"]`. OpenSearch supports column filtering and reordering. For example, if a table has fields a, b, and c, and you need only c and b, configure `["c","b"]`. During import, field a is populated with null.	Yes	None
batchSize	Number of records written per batch. OpenSearch supports batch writes and generally excels at queries rather than high write throughput. Set this based on the resources allocated to your account. Typically, a single OpenSearch record is smaller than 1 MB and a batch write is smaller than 2 MB.	This option is required for partitioned tables and must be omitted for non-partitioned tables.	300