OSS + API data source

---

This topic describes how to add a table that uses Object Storage Service (OSS) + API as a data source.

To add an OSS + API data source, follow these two steps:

1. Activate Object Storage Service (OSS).

2. Add the OSS + API data source.

---

Step 1: Activate OSS

1. Activate OSS

The Object Storage Service (OSS) you activate must be under the same account and in the same region as your OpenSearch instance.

On the OSS resource plan purchase page, set Product Type to OSS Resource Plan. Select a Resource Plan Type, such as Standard - Locally Redundant Storage, based on your business needs and choose a corresponding region, such as Chinese mainland.

2. Create a bucket

You must create a bucket for your data files before uploading them to OSS. For more information, see Create buckets.

Path: Object Storage Service (OSS) console → Buckets → Create Bucket

Navigate to the bucket you created and add an OpenSearch tag.

Path: Bucket Configuration → Bucket Tagging → Add Tag

Set both the tag key and value to opensearch.

3. OSS file format

a. OSS file format

To be used as a data source for an index, your files must meet the following requirements:

• Files must be UTF-8 encoded.

• Data must be formatted as either HA3 or JSON.

i. JSON format

The following code shows an example with multiple records:

{"field_double": ["100.0", "221.123", "500.3333333"], "field_int32": ["100", "200", "300"], "title": "Huawei Mate 9 Kirin 960 Chip Leica Dual Lens", "color": "Red", "empty_int32": "", "price": "3599", "CMD": "add", "nid": "1", "gather_cn_str": "", "desc": ["str1", "str2", "str3"], "brand": "Huawei", "size": "5.9","__subdocs__":[{"sub_pk":"100","sub_field1":"200","sub_field2":["100","200","300"]},{"sub_pk":"200","sub_field1":"200","sub_field2":["100","200","300"]}]}
{"field_double": ["100.0", "221.123", "500.3333333", "100.0", "221.123", "500.3333333"], "field_int32": ["100", "200", "300", "100", "200", "300"], "title": "Huawei P10 Plus Unlocked Phone", "color": "Blue", "empty_int32": "", "price": "4388", "CMD": "add", "nid": "2", "gather_cn_str": "colorBlue", "desc": ["str1", "str2", "str3", "str1", "str2", "str3"], "brand": "Huawei", "size": "5.5","__subdocs__":[{"sub_pk":"100","sub_field1":"200","sub_field2":["100","200","300"]},{"sub_pk":"200","sub_field1":"200","sub_field2":["100","200","300"]}]}

Notes:

• All field values in the JSON data must be strings. The search engine converts these strings to the data types specified in the index schema during indexing.

• Each JSON object represents a single record and must not contain line breaks. Records must be separated by a newline character (\n).

ii. HA3 format

1. The following code shows the content of a sample data file, standard_sample.data.

CMD=add^_
PK=12345321^_
url=http://www.aliyun.com/index.html^_
title=Alibaba Cloud Computing Co., Ltd.^_
body=xxxxxx xxx^_
time=3123423421^_
multi_value_field=1234^]324^]342^_
bidwords=mp3^\price=35.8^Ptime=13867236221^]mp4^\price=32.8^Ptime=13867236221^_
^^
CMD=delete^_
PK=12345321^_CMD=add^_
PK=12345321^_
url=http://www.aliyun.com/index.html^_
title=Alibaba Cloud Computing Co., Ltd.^_
body=xxxxxx xxx^_
time=3123423421^_
multi_value_field=1234^]324^]342^_
bidwords=mp3^\price=35.8^Ptime=13867236221^]mp4^\price=32.8^Ptime=13867236221^_
^^
CMD=delete^_
PK=12345321^_

2. Delimiters: The data file contains add and delete commands. Each command consists of key-value pairs on separate lines. Commands are separated by ^^\n, key-value pairs are separated by ^_\n, and multiple values within a field are separated by ^]. The following table describes the delimiters.

C++ encoding	ASCII Hex	Description	Display in emacs/vi	Input method in emacs	Input method in vi
"\x1F\n"	1F0A	Key-value delimiter	^_ (followed by a newline)	C-q C-7	C-v C-7
"\x1E\n"	1E0A	Command delimiter	^^ (followed by a newline)	C-q C-6	C-v C-6
"\x1D"	1D	Multi-value delimiter	^]	C-q C-5	C-v C-5
"\x1C"	1C	Section weight identifier	^\	C-q C-4	C-v C-4
"\x1D"	1D	Section delimiter	^]	C-q C-5	C-v C-5
"\x03"	03	Sub-doc field delimiter	^C	C-q C-c	C-v C-c

3. add command format: Adds a new document to the index. The command must start with CMD=add on the first line, followed by the document's fields. The field order can differ from the schema, but all fields must be defined in the schema.

CMD=add^_
PK=12345321^_
url=http://www.aliyun.com/index.html^_
title=Alibaba Cloud Computing Co., Ltd.^_
body=xxxxxx xxx^_
time=3123423421^_
multi_value_field=1234^]324^]342^_
bidwords=mp3^\price=35.8^Ptime=13867236221^]mp4^\price=32.8^Ptime=13867236221^_
^^CMD=add^_
PK=12345321^_
url=http://www.aliyun.com/index.html^_
title=Alibaba Cloud Computing Co., Ltd.^_
body=xxxxxx xxx^_
time=3123423421^_
multi_value_field=1234^]324^]342^_
bidwords=mp3^\price=35.8^Ptime=13867236221^]mp4^\price=32.8^Ptime=13867236221^_
^^

4. delete command format: Removes a document from the index. The command must start with CMD=delete on the first line, followed by the primary key defined in the index schema and the field used for partition hashing. If the primary key and hash field are the same, only one is required.

CMD=delete^_
PK=12345321^_
^^CMD=delete^_
PK=12345321^_
^^

b. Upload files to the bucket

Return to the console page of your bucket. In the left-side navigation pane, choose File Management → File List → Upload File.

In the Upload panel, select your files, and then click Upload File.

In the Tasks panel that appears on the right, confirm that the file status shows Succeeded.

Step 2: Add OSS + API data source

You must have a running instance. For instructions, see Purchase an OpenSearch Retrieval Engine Edition instance.

1. Basic table information

From the Instance List, find your Retrieval Engine Edition instance and click Manage in the Actions column.

In the left-side navigation pane, choose Table Management, and then click Add Table.

Enter the basic information for your table, and then click Next.

• Table Name: Enter a name that contains only letters, digits, or underscores, up to 25 characters.

• Number of Data Shards: Enter a positive integer up to 256. Using more data shards can improve full indexing speed and query performance. For some existing instances, tables must adhere to one of two sharding rules: either all tables have the same number of data shards, or one table has a single shard while all others share a different, identical number of shards.

• Number of Data Update Resources: The number of resources allocated for data updates. By default, two free data update resources (4 vCPUs, 8 GiB memory) are provided for each index. You are charged for any resources that exceed this free quota. For more information, see Billing overview for Retrieval Engine Edition.

2. Data synchronization

For Full Data Source, select "Object Storage Service (OSS) + API". Configure the other parameters, and after the Data Source Verification passes, click Next.

Parameters:

• OSS Path: The path to access the OSS files. It must start with a forward slash (/) and cannot contain ?, =, or &. Files cannot be in the root directory; they must be in a subfolder.

• OSS bucket: The name of the OSS bucket.

In the Object Storage Service (OSS) console, on the Buckets page, click the name of your target bucket (for example, zhyq-sz) to go to its management page.

• Data Format: Data files must be in either HA3 format or JSON format to be ingested.

• Data Source Verification: After the verification passes, you can proceed to the next step.

3. Index schema

Configure the fields and indexes for your table, then click Next.

Example field configuration: id (INT64, primary key, attribute field, equal data compression), fb_string (STRING, multi-value with comma delimiter, uniq data compression), name (TEXT, Chinese - General analyzer), and code (STRING, attribute field, uniq data compression, custom default value of <EMPTY_STRING>).

Example index configuration: Add an index named index_name (type TEXT, includes the name field) and an index named index_string (type STRING, includes the fb_string field).

• Configuration details:

  • Field settings:

    • Attribute Field: You can choose whether to compress attribute fields. Compression is disabled by default. Select file_compressor to enable compression.

    • Data Compression: Reduces storage by de-duplicating field content. You can choose whether to enable compression. Compression is disabled by default. The recommended compression type is uniq for STRING and multi-value types, and equal for single-value numeric types.

    • Usage notes:

      1. If you enable attribute compression, we recommend that you edit the index loading method in Cluster Management > Node Status > Table Loading Policy to mitigate any impact on performance.

      2. Only fields of the TEXT type can be configured with an analyzer.

      3. You can copy fields. A copied field (DUP field) has the same content as its source field. If you push different content to the DUP field, the source field's content will overwrite it. To maintain different content, manually remove the copy_from configuration in the DUP field's advanced settings.

      4. If a field is missing or empty in your source data, the system automatically inserts a default value. The default is 0 for numeric types and an empty string for STRING types. You can also specify a custom default value.

  • Index settings: Create indexes for searchable fields.

4. Create the table

Click Confirm Creation. This creates the table based on your configuration.