Description of data source plug-ins
The system provides several data processing plug-ins that you can use to perform simple data conversions. This topic describes the data source plug-ins.
Description
You can import data into OpenSearch by using the API, SDKs, or the upload page in the console. You can also synchronize data directly from an existing cloud database. If you upload data by using the API or SDKs, you can upload the data as described in the API reference. In this case, the plug-in configurations described in this topic are not supported. If you synchronize data from a cloud database, you must configure the data source in the console. The system provides several data processing plug-ins that you can use to perform simple data conversions. You can use these plug-ins when you configure field mappings for a data source. Plug-ins are not supported for data uploaded by using the API. You must process such data before you push it.
An OpenSearch table supports multiple ApsaraDB RDS and PolarDB source tables, such as in database and table sharding scenarios. However, you can configure only one MaxCompute (ODPS) data source. If you need data from multiple MaxCompute (ODPS) source tables, merge the data into a single table before you import it.
Data processing plug-ins
Some search features or feature functions require special field types. For example, fields of the ARRAY type must be converted by using the following plug-ins. You cannot enter such values directly.
These plug-ins are configured when you configure a data source, not when you define the application schema. You can configure plug-ins only after you configure a data source.
Configuration item | Description | Example |
JsonKeyValueExtractor | Extracts the value of a specified key from a source field in the JSON format. The extracted value is used as the content of the destination table field. You can extract only the value of a single key. | Extract the content of the title key from {“title”:”the content”,”body”:”the content”}. If the content is in the JsonArray format, it is converted into the content of an ARRAY field in the system. “Make sure that the type of the extracted value matches the type of the destination table field. Otherwise, the corresponding data is lost.” The JsonArray format here refers to the JsonArray format defined by OpenSearch. Examples: for a literal_array field, {“tags”:[“a”,”b”,”c”]}; for an int_array field, {“tags”:[1,2,3]}. |
MultiValueSpliter | Splits a source field into multiple values based on a delimiter. The split content is used as the content of the destination table field. The destination table field must be configured as an ARRAY field. If the delimiter is an invisible character, you must specify it as a Unicode character, such as \u001D. | If the content of the data source is 1,2,3, specify the delimiter as “,”. Simply enter a half-width (English) comma. |
KeyValueExtractor | Extracts the value of a specified key from a source field in the key-value (KV) format. The extracted value is used as the content of the destination table field. You can extract only the value of a single key. The delimiters are optional. | The actual content is key1:value1,value2;key2:value3. The keys are key1 and key2, the key delimiter is a semicolon, the key-value delimiter is a colon, and the multi-value delimiter is a comma. If a multi-value delimiter is configured, the content is converted into the content of an ARRAY field in the system. “Make sure that the type of the extracted value matches the type of the destination table field. Otherwise, the corresponding data is lost.” If two identical keys exist, only the value of the latter key is extracted. |
StringCatenateExtractor | Concatenates multiple specified fields into a single string in the specified order. This plug-in does not support fields of the INT type. We recommend that you use fields of the LITERAL type. Separate fields in the field list with commas. The fields must be destination fields. | Concatenates the content of field1 and field2 with an underscore (‘_’) to form the content of a new field. In addition, the system variable $table can be used to obtain the name of the current table. ($table is displayed only when a wildcard for sharded tables is configured.) |
HTMLTagRemover | Removes HTML tags from a source field and copies the content without tags to the current field. If the source field is configured as the current field, the content is replaced. | Original text: < div id=”copyright”>OpenSearch< /div>. After the plug-in is used, the text is parsed as “OpenSearch”. |
For best practices, see Configure the MultiValueSpliter plugin.