OSS + API data source

The procedure consists of the following two steps:

1. Activate OSS

2. Add an OSS + API data source

---

Step 1: Activate OSS

1. Activate the service

Activate OSS. OSS must use the same account and be in the same region as your purchased OpenSearch service.

On the OSS resource plan purchase page, set Product Type to OSS Resource Plan, Resource Plan Type to Standard - Zone-redundant Storage (ZRS), and Region to Chinese Mainland General. The region must be the same as the region of your purchased OpenSearch service.

2. Create a bucket

Before you upload files to OSS, you must create a bucket to store the files. For more information, see Create a bucket.

Path: OSS console → Buckets → Create Bucket

In the left-side navigation pane of the OSS console, click Buckets, and then click Create Bucket.

Go to the bucket that you created and add an OpenSearch tag.

Path: Bucket Settings → Bucket Tagging → Create Tag

Set both the tag key and the tag value to opensearch.

3. Configure the OSS file format

a. OSS file format

To ensure that the uploaded files can be used as a data source for index building, make sure that:

• The files are encoded in UTF-8.

• The file data is converted to the HA3 format or the JSON format.

i. JSON format

For multiple records, see the following example:

{"field_double": ["100.0", "221.123", "500.3333333"], "field_int32": ["100", "200", "300"], "title": "Huawei Mate 9 Kirin 960 chip Leica dual camera", "color": "red", "empty_int32": "", "price": "3599", "CMD": "add", "nid": "1", "gather_cn_str": "", "desc": ["str1", "str2", "str3"], "brand": "Huawei", "size": "5.9","__subdocs__":[{"sub_pk":"100","sub_field1":"200","sub_field2":["100","200","300"]},{"sub_pk":"200","sub_field1":"200","sub_field2":["100","200","300"]}]}
{"field_double": ["100.0", "221.123", "500.3333333", "100.0", "221.123", "500.3333333"], "field_int32": ["100", "200", "300", "100", "200", "300"], "title": "Huawei P10 Plus all-network phone", "color": "blue", "empty_int32": "", "price": "4388", "CMD": "add", "nid": "2", "gather_cn_str": "colorblue", "desc": ["str1", "str2", "str3", "str1", "str2", "str3"], "brand": "Huawei", "size": "5.5","__subdocs__":[{"sub_pk":"100","sub_field1":"200","sub_field2":["100","200","300"]},{"sub_pk":"200","sub_field1":"200","sub_field2":["100","200","300"]}]}

ii. HA3 format

1. First, view the content of a complete data file named standard_sample.data.

CMD=add^_
PK=12345321^_
url=http://www.aliyun.com/index.html^_
title=Alibaba Cloud Computing Co., Ltd.^_
body=xxxxxx xxx^_
time=3123423421^_
multi_value_field=1234^]324^]342^_
bidwords=mp3^\price=35.8^Ptime=13867236221^]mp4^\price=32.8^Ptime=13867236221^_
^^
CMD=delete^_
PK=12345321^_CMD=add^_
PK=12345321^_
url=http://www.aliyun.com/index.html^_
title=Alibaba Cloud Computing Co., Ltd.^_
body=xxxxxx xxx^_
time=3123423421^_
multi_value_field=1234^]324^]342^_
bidwords=mp3^\price=35.8^Ptime=13867236221^]mp4^\price=32.8^Ptime=13867236221^_
^^
CMD=delete^_
PK=12345321^_

2. File separator definitions: The preceding data file contains two commands: add and delete. Each command consists of multiple lines, and each line is a key-value pair. Commands are separated by '^^\n', key-value pairs are separated by '^_\n', and multiple values are separated by '^]'. The following table describes the file separators.

C++ encoding	ASCII Hexadecimal	Description	Display form (in emacs/vi)	Input method in emacs	Input method in vi
"\x1F\n"	1F0A	Key-value separator	^_ (followed by a line feed)	C-q C-7	C-v C-7
"\x1E\n"	1E0A	Command separator	^^ (followed by a line feed)	C-q C-6	C-v C-6
"\x1D"	1D	Multi-value separator	^]	C-q C-5	C-v C-5
"\x1C"	1C	Section weight identifier	^\	C-q C-4	C-v C-4
"\x1D"	1D	Section separator	^]	C-q C-5	C-v C-5
"\x03"	03	Sub doc field separator	^C	C-q C-c	C-v C-c

3. Command format definitions

   Add command format: The add command adds new content to the index. The first line of an add command must be CMD=add, followed by the fields of the document. The order of the fields can be the same as the order of the fields in the schema. All fields that appear must be specified in fields.

   CMD=add^_
   PK=12345321^_
   url=http://www.aliyun.com/index.html^_
   title=Alibaba Cloud Computing Co., Ltd.^_
   body=xxxxxx xxx^_
   time=3123423421^_
   multi_value_field=1234^]324^]342^_
   bidwords=mp3^\price=35.8^Ptime=13867236221^]mp4^\price=32.8^Ptime=13867236221^_
   ^^CMD=add^_
   PK=12345321^_
   url=http://www.aliyun.com/index.html^_
   title=Alibaba Cloud Computing Co., Ltd.^_
   body=xxxxxx xxx^_
   time=3123423421^_
   multi_value_field=1234^]324^]342^_
   bidwords=mp3^\price=35.8^Ptime=13867236221^]mp4^\price=32.8^Ptime=13867236221^_
   ^^

   Delete command format: The delete command deletes specified content from the index. The first line of a delete command must be CMD=delete, followed by the field defined as the primary key in the index schema and the field used for partition hashing. If the two fields are the same, the field needs to appear only once.

   CMD=delete^_
   PK=12345321^_
   ^^CMD=delete^_
   PK=12345321^_
   ^^

b. Upload files to the bucket

Return to the management page of the bucket that you created. In the left-side navigation pane, choose Object Management → Objects → Upload Object .

In the Files to Upload section, select a method to scan files, select the content to upload, and then click Upload Object to complete the upload.

After the upload is complete, the file status in the Task List panel is displayed as Uploaded.

Step 2: Add an OSS + API data source

Purchase an instance: For more information, see Purchase an OpenSearch Vector Search Edition instance.

1. Basic table information

On the Instances page, find the OpenSearch Vector Search Edition instance that you purchased and click Manage in the Actions column. In the left-side navigation pane, choose Table Management → Add Table .

After you specify the basic information of the table, click Next.

After you select the Vector: Image Search template, you must select a retrieval mode: Dense Vector Retrieval or Dense & Sparse Vector Hybrid Retrieval.

Configuration description:

• Table name: You can customize the name.

• Number of data shards: Enter a positive integer that does not exceed 256. Shards are used to improve full data building speed and single-query performance.

• Number of data update resources: the number of resources used for data updates. By default, each index is provided with two free update resources, each with 4 cores and 8 GB of memory. Resources that exceed the free quota incur fees. For more information, see Billing overview of OpenSearch Vector Search Edition.

• Scenario template: Three templates are available: 1. General template, 2. Vector: Image Search template, and 3. Vector: Text Semantic Search template.

2. Data synchronization

Set Full Data Source to "Object Storage Service (OSS) + API" and configure the other parameters. After the Data Source Verification succeeds, click Next .

Configuration description:

• OSS Path: the path used to access the OSS files. The path must start with a forward slash (/) and cannot contain the ?, =, or & characters. The files cannot be stored in the root directory. Store them in a folder and enter the path of the folder.

• OSS Bucket: the name of the OSS bucket.

On the Buckets page of the OSS console, click the name of the target bucket (for example, zhyq-sz) to go to the bucket details page.

• Data Format: The files must be in the HA3 or JSON format. Otherwise, the file data cannot be transferred.

• Data Source Verification: You can proceed to the next step after the verification succeeds.

3. Field configuration

OpenSearch Vector Search Edition presets related fields based on the scenario template that you selected and automatically imports the fields from the full data source into the list below. After you complete the Field Configuration and Data Preprocessing Required - Configure settings, click Next .

Field descriptions:

• id (primary key)

• source_image (source image)

• namespace (namespace)

• source_image_vector (source image vector)

Configuration description:

• Required fields are the primary key field and the vector field. The primary key field must be of the int or string type, and you must select the primary key option for it. The vector field must be of the float type, and you must select the vector field option for it.

• By default, the vector field is a multi-value field of the float type, and multiple values are split by commas (,). You can also enter a custom multi-value separator.

• If a field is missing from the data or a field is empty, the system automatically fills in a default value: 0 for numeric types and an empty string for the STRING type. Custom default values are supported.

• Data Preprocessing Required - Configure: Click Configure to go to the data preprocessing configuration page of the source_image field.

  Configuration values used in this example:

  • OSS Endpoint: oss-cn-shenzhen-internal.aliyuncs.com

  • OSS Bucket: zhyq-sz

  • In the service list, select the Built-in model General Image-to-Vector Model - 512-dimension CLIP. The generated field is source_image_vector.

  1. Data source: Two data type options are available: OSS and Base64 encoding. In this example, OSS is selected.

     • OSS: You must enter an OSS path. The images are stored in an OSS folder and imported directly from OSS.

     • Base64 encoding: The images must first be encoded and then stored in a database, or transferred directly by using the API.

  2. Preprocessing template: Different templates are displayed based on the type of data to preprocess (text or image). Because image data is preprocessed in this example, three preprocessing templates are displayed: 1. Image vectorization, 2. OCR image text recognition, and 3. OCR image text recognition + image vectorization. In this example, the Image Embedding preprocessing template is selected.

  3. Service list:

     • After you select a preprocessing template, the service list of the template automatically appears and shows the types of models used by the template.

     • Available model sources:

       • Built-in models: A limited number and variety of models are available, and they can be called free of charge.

       • AI Search Open Platform: If you select the OCR image text recognition template or the OCR image text recognition + image vectorization template, the AI Search Open Platform model option appears. AI Search Open Platform models provide rich capabilities and are billed based on the number of calls. For more information, see Billing methods and billable items of AI Search Open Platform. Before you use these models, you must activate a workspace and an API key on AI Search Open Platform.

       • Custom models: You can customize models based on your requirements. On the OpenSearch Vector Search Edition page, choose Models>Custom Models and perform the Create Model operation. For more information, see Custom models.

4. Index schema

Configure the index schema. After the configuration is complete, click Next.

On the Vector Index tab, the parameters in this example are set as follows:

• Index Name: source_image_vector

• Hybrid Retrieval: Disabled

• Vector Dimensions: 512

• Distance Type: SquaredEuclidean

• Real-time Index: true

• Vector Index Algorithm: Qc

Configuration description:

• Included fields: The primary key field and the vector field are required. The namespace field is optional and can be left empty.

• You can specify the vector dimensions, real-time index, distance type, and vector index algorithm based on your business requirements.

• More advanced settings: You can use the default parameters or adjust them based on your business requirements. For more information, see General vector index configuration. After you complete the settings on the Index Schema page, click Next to go to the Confirm page.

The advanced settings include the following parameters:

• Real-time Index Parameters

• Index Building Parameters

• Index Retrieval Parameters

• Linear Building Threshold

• Minimum Number of Recall Candidates

• Recall Rate Metric Reporting

• Ignore Invalid Vector Data

5. Confirm

Click Confirm . After the creation is complete, wait 2 minutes and return to the Instances page. After the instance status changes to "Normal ", you can choose Actions - Query Test to perform subsequent searches and tests.