Data structures of a DataSource-OpenSearch(Open Search)-阿里云帮助中心

Describes the configuration of a data source.

Example

{
  "tableName": "news",
  "type": "rds",
  "fields": [
    {
      "id": "id"
    },
    {
      "caption": "caption"
    },
    {
      "content": "content"
    },
    {
      "categoryId": "category_id"
    }
  ],
  "plugins": {
    "caption": {
      "name": "HTMLTagRemover",
      "parameters": {}
    },
    "content": {
      "name": "HTMLTagRemover",
      "parameters": {}
    }
  },
  "keyField": "id",
  "parameters": {
    "filter": "",
    "instanceId": "rm-abc",
    "dbName": "my_db",
    "dbTableName": "my_table",
    "dbUser": "my_user",
    "dbPassword": "my-password",
    "autoSync": true
  }
}

Structure

Field	Type	Description
tableName	String	The name of the table.
type	String	The type of the data source. Valid values: rds, odps, opensearch, and polardb.
fields[]	Object	The field mappings. Each mapping uses the format `{key:value}`, where `key` is the field from the source table and `value` is the field in the destination table of the application.
plugins	Object	The plug-ins used for data processing.
plugins.*	Object	The configuration of a plug-in. For more information, see the Plugin section. The asterisk (`*`) indicates the name of the plug-in.
keyField	String	The primary key.
parameters	Object	The parameters for the data source. Parameters vary based on the data source type. For more information, see the following sections: rds, odps, opensearch, polardb.

Plugin

Example

{
  "name": "HTMLTagRemover",
  "parameters": {}
}

Structure

Field	Type	Description
name	String	The name of the plug-in. Valid values: JsonKeyValueExtractor, MultiValueSpliter, KeyValueExtractor, StringCatenateExtractor, and HTMLTagRemover.
parameters	Object	The parameters for the plug-in. Parameters vary based on the plug-in. For more information, see the following sections: JsonKeyValueExtractor, MultiValueSpliter, KeyValueExtractor, StringCatenateExtractor, HTMLTagRemover.

JsonKeyValueExtractor

Extracts the value of the specified key from source fields in JSON format. The extracted key value is used as the value of the mapped destination field. Only the value of the specified key can be extracted.

Example

{
  "key": "my_field"
}

Structure

Field	Type	Description
key	String	The name of the key.

MultiValueSpliter

Uses delimiters to divide the value of a source field into multiple values. The values are used as the array elements of the mapped destination field. The destination field must be of the ARRAY type. If you use a non-printable character as the delimiter, use Unicode characters such as \u001D to identify the non-printable character.

Example

{
  "seperator": "|"
}

Structure

Field	Type	Description
seperator	String	The delimiter.

KeyValueExtractor

Extracts the value of the specified key from source fields that are key-value pairs. The extracted key value is used as the value of the mapped destination field. Only the value of the specified key can be extracted. Delimiters are optional.

Example

{
  "key": "my_field",
  "fieldSep": ";",
  "kvSep": ":"
}

Structure

Field	Type	Description
key	String	The name of the key.
fieldSep	String	The delimiter used to separate keys.
kvSep	String	The delimiter used to separate keys and values.

StringCatenateExtractor

Concatenates the values of specified fields into a string in the specified sequence. This plug-in cannot concatenate fields of integer types. We recommend that you use fields of literal types. Separate multiple fields with commas (,). The fields must be destination fields.

Example

{
  "connector": "-",
  "sources": "id,title"
}

Structure

Field	Type	Description
connector	String	The connector.
sources	String	The fields to concatenate.

HTMLTagRemover

Removes HTML tags from the value of a source field. The value of the mapped destination field is replaced by the value without HTML tags.

This plug-in does not require additional parameters.

rds

Example

{
  "instanceId": "rds-instance-id",
  "dbName": "my_db",
  "dbTableName": "my_table",
  "dbUser": "my",
  "dbPassword": "my_passwd",
  "filter": "",
  "autoSync": true
}

Structure

Field	Type	Description
instanceId	String	The ID of the instance.
dbName	String	The name of the database.
dbTableName	String	The name of the table in the database.
dbUser	String	The account used to connect to the database.
dbPassword	String	The password used to connect to the database.
filter	String	The filter conditions. Supported operators: `<`, `>`, `=`, `!=`, `>=`, and `<=`. Separate multiple conditions with commas (,).
autoSync	Boolean	Specifies whether automatic data synchronization is enabled. Valid values: true and false.

odps

Example

{
  "projectName": "my_project",
  "tableName": "my_table",
  "partition": "",
  "done": "no",
  "donePrefix": "",
  "timestamp": ""
}

Structure

Field	Type	Description
projectName	String	The name of the MaxCompute project.
tableName	String	The name of the table.
partition	String	The partition. For more information, see Configure a MaxCompute data source.
done	String	Specifies whether the DONE file is used. Valid values: no and yes.
donePrefix	String	The prefix of the DONE file. Required if `done` is set to yes.
timestamp	String	The timestamp when full data was imported.

opensearch

Example

{
  "appId": "12345678",
  "tableName": "my_table",
  "appType": "standard",
  "filter": ""
}

Structure

Field	Type	Description
appId	String	The ID of the application.
tableName	String	The name of the table in the application.
appType	String	The type of the application. Valid values: standard, advance, and enhanced. standard indicates a standard application. advance indicates an advanced application of a previous application type. enhanced indicates an advanced application of a new application type. New applications cannot be of the advance type.
filter	String	The filter conditions. Supported operators: `<`, `>`, `=`, `!=`, `>=`, and `<=`. Separate multiple conditions with commas (,).

polardb

Example

{
  "clusterId": "my_cluster_id",
  "dbName": "my_db",
  "dbTableName": "my_table",
  "aliyunUserId": "123455789",
  "dbUser": "my_user",
  "dbPassword": "my_passwd",
  "autoSync": true,
  "filter": "id>100"
}

Structure

Field	Type	Description
clusterId	String	The ID of the cluster.
dbName	String	The name of the database.
dbTableName	String	The name of the table.
aliyunUserId	String	The ID of the Alibaba Cloud account.
dbUser	String	The account used to connect to the database.
dbPassword	String	The password used to connect to the database.
autoSync	Boolean	Specifies whether automatic data synchronization is enabled. Valid values: true and false.
filter	String	The filter conditions. Supported operators: `<`, `>`, `=`, `!=`, `>=`, and `<=`. Separate multiple conditions with commas (,).