Fields in the DataSource data structure-OpenSearch(Open Search)-阿里云帮助中心

Description

Data source information

Example

{
  "tableName": "news",
  "type": "rds",
  "fields": [
    { "id": "id" },
    { "caption": "caption" },
    { "content": "content" },
    { "categoryId": "category_id" }
  ],
  "plugins": {
    "caption": {
      "name": "HTMLTagRemover",
      "parameters": {}
    },
    "content": {
      "name": "HTMLTagRemover",
      "parameters": {}
    }
  },
  "keyField": "id",
  "parameters": {
    "filter": "",
    "instanceId": "rm-abc",
    "dbName": "my_db",
    "dbTableName": "my_table",
    "dbUser": "my_user",
    "dbPassword": "my-password",
    "autoSync": true
  }
}

Structure

Field	Type	Description
`tableName`	String	The name of the table in the source system.
`type`	String	The type of the data source. Valid values: `rds`, `odps`, `opensearch`, and `polardb`. The value determines which fields are required in `parameters`.
`fields[]`	Object	Field mappings between the source table and the destination table. Each entry is a `{key: value}` pair where `key` is the source field name and `value` is the destination field name.
`plugins`	Object	Data processing plug-ins applied to specific fields before indexing. Each key is a destination field name; each value is a Plugin object.
`plugins.*`	Object	Plug-in details. For more information, see Plugin below. *: field name.
`keyField`	String	The primary key of the data source table.
`parameters`	Object	Connection and sync parameters for the data source. The required fields depend on the value of `type`. See rds, odps, opensearch, and polardb.

Plugin

Example

{
  "name": "HTMLTagRemover",
  "parameters": {}
}

Structure

Field	Type	Description
`name`	String	The plug-in to apply. Valid values: `JsonKeyValueExtractor`, `MultiValueSpliter`, `KeyValueExtractor`, `StringCatenateExtractor`, and `HTMLTagRemover`.
`parameters`	Object	Plug-in configuration. The required fields depend on the value of `name`. See the plug-in sections below.

JsonKeyValueExtractor

Extracts the value of a specified key from a source field that contains JSON. The extracted value is written to the destination field. Only one key can be extracted per plug-in instance.

Example

{
  "key": "my_field"
}

Structure

Field	Type	Description
`key`	String	The key to extract from the JSON source field.

MultiValueSpliter

Splits a source field value into multiple values using a delimiter. The resulting values are written as array elements to the destination field.

The destination field must be of the ARRAY type. To use a non-printable character as the delimiter, specify it as a Unicode escape sequence (for example, \u001D).

Example

{
  "seperator": "|"
}

Structure

Field	Type	Description
`seperator`	String	The delimiter used to split the source field value.

KeyValueExtractor

Extracts the value of a specified key from a source field that contains key-value pairs. The extracted value is written to the destination field. Delimiters are optional.

Example

{
  "key": "my_field",
  "fieldSep": ";",
  "kvSep": ":"
}

Structure

Field	Type	Description
`key`	String	The key whose value to extract.
`fieldSep`	String	The delimiter that separates multiple key-value pairs.
`kvSep`	String	The delimiter that separates a key from its value.

StringCatenateExtractor

Concatenates the values of multiple destination fields into a single string using a specified connector, and writes the result to the target field.

This plug-in cannot concatenate fields of integer types. We recommend that you use fields of literal types. All fields listed in sources must be destination fields.

Example

{
  "connector": "-",
  "sources": "id,title"
}

Structure

Field	Type	Description
`connector`	String	The string inserted between concatenated field values.
`sources`	String	A comma-separated list of destination field names to concatenate.

HTMLTagRemover

Strips HTML tags from the value of a source field and writes the plain-text result to the destination field. No additional parameters are required.

rds

Connects to an ApsaraDB RDS instance.

Example

{
  "instanceId": "rds-instance-id",
  "dbName": "my_db",
  "dbTableName": "my_table",
  "dbUser": "my_user",
  "dbPassword": "my_passwd",
  "filter": "",
  "autoSync": true
}

Structure

Field	Type	Description
`instanceId`	String	The ID of the RDS instance.
`dbName`	String	The name of the database.
`dbTableName`	String	The name of the table in the database.
`dbUser`	String	The account used to connect to the database.
`dbPassword`	String	The password used to connect to the database.
`filter`	String	Filter conditions that limit which rows are synchronized. Supported operators: `<`, `>`, `=`, `!=`, `>=`, and `<=`. Separate multiple conditions with commas (,).
`autoSync`	Boolean	Specifies whether to enable automatic data synchronization. Valid values: `true` and `false`.

odps

Connects to a MaxCompute (ODPS) table.

Example

{
  "projectName": "my_project",
  "tableName": "my_table",
  "partition": "",
  "done": "no",
  "donePrefix": "",
  "timestamp": ""
}

Structure

Field	Type	Description
`projectName`	String	The name of the MaxCompute project.
`tableName`	String	The name of the table.
`partition`	String	The partition to read from. For partition syntax, see Configure a MaxCompute data source.
`done`	String	Specifies whether to use a DONE file to signal that a data export is complete. Valid values: `no` and `yes`.
`donePrefix`	String	The prefix of the DONE file. Required when `done` is set to `yes`.
`timestamp`	String	The timestamp of the full data import.

opensearch

Uses another Open Search application as the data source.

Example

{
  "appId": "12345678",
  "tableName": "my_table",
  "appType": "standard",
  "filter": ""
}

Structure

Field	Type	Description
`appId`	String	The ID of the source Open Search application.
`tableName`	String	The name of the table in the source application.
`appType`	String	The type of the source application. Valid values: `standard`, `advance`, and `enhanced`. `standard` indicates a standard application. `advance` indicates an advanced application of a previous generation. `enhanced` indicates an advanced application of the current generation. New applications cannot use the `advance` type.
`filter`	String	Filter conditions that limit which records are synchronized. Supported operators: `<`, `>`, `=`, `!=`, `>=`, and `<=`. Separate multiple conditions with commas (,).

polardb

Connects to a PolarDB cluster.

Example

{
  "clusterId": "my_cluster_id",
  "dbName": "my_db",
  "dbTableName": "my_table",
  "aliyunUserId": "123455789",
  "dbUser": "my_user",
  "dbPassword": "my_passwd",
  "autoSync": true,
  "filter": "id>100"
}

Structure

Field	Type	Description
`clusterId`	String	The ID of the PolarDB cluster.
`dbName`	String	The name of the database.
`dbTableName`	String	The name of the table.
`aliyunUserId`	String	The ID of the Alibaba Cloud account.
`dbUser`	String	The account used to connect to the database.
`dbPassword`	String	The password used to connect to the database.
`autoSync`	Boolean	Specifies whether to enable automatic data synchronization. Valid values: `true` and `false`.
`filter`	String	Filter conditions that limit which rows are synchronized. Supported operators: `<`, `>`, `=`, `!=`, `>=`, and `<=`. Separate multiple conditions with commas (,).