DocsICP FilingConsole
官方文档
首页

Quick start for text search

更新时间：
复制 MD 格式
产品详情
我的收藏

This topic describes how to get started with text search using a traditional instance of OpenSearch Vector Search Edition.

Important

  • This topic applies only to traditional instances of OpenSearch Vector Search Edition. The database engine version of these instances starts with `ha3`, such as `ha3_3.10.0`. These instances are no longer available for purchase.

  • The instances that are currently available for purchase are easy-to-use instances of OpenSearch Vector Search Edition. The database engine version of these instances starts with `vector_service`, such as `vector_service_1.4.2`. For more information, see Quick start for text search in the easy-to-use edition.

Prerequisites

  1. You have registered an Alibaba Cloud account and completed identity verification.

  2. When you log on to the console with your Alibaba Cloud account for the first time, you are prompted to create an AccessKey pair.

    1. An AccessKey pair is required to create and use applications. You must create an AccessKey pair for your Alibaba Cloud account.

    2. After you create an AccessKey pair for your Alibaba Cloud account, you can create an AccessKey pair for a RAM user and use the RAM user to access OpenSearch. For more information about how to grant permissions to a RAM user, see Authorization rules . Note: The RAM user must be granted the `AliyunSearchEngineFullAccess` or `AliyunSearchEngineReadOnlyAccess` permission to access instances of OpenSearch Vector Search Edition.

  3. A VPC environment is required. For more information, see Create a VPC.

Purchase an instance

  1. Log on to the OpenSearch console and select OpenSearch-Vector Search Edition in the upper-left corner.

  1. On the Instance Management page of the Vector Search Edition console, click Create Instance.

  1. Set Product Version to Vector Search Edition. Select a region. Configure the number and specifications for query nodes and data nodes. Set the total storage space for a single data node. Configure the VPC and Virtual Switch. Set a username and password for query authentication. This password is not your Alibaba Cloud account password. Then, click Buy Now.

Note

  • Plan the number and specifications of query nodes and data nodes based on your requirements. After you specify the specifications, the actual fee is automatically calculated on the buy page.

  • The VPC and virtual switch must be the same as those of the ECS instance that you use to access the OpenSearch Vector Search Edition instance. Otherwise, the `{'errors':{'code':'403','message':'Forbidden'}}` error is returned.

  • Each data node comes with a free storage quota. You can also request additional storage, which is billed in 50 GB increments.

  1. On the Confirm Order page, review the Terms of Service. If the order details are correct, click Activate Now.

  1. After you complete the purchase, click Management Console to view the new instance on the Instance Management page.

  1. A new instance is assigned a default name. To change the name, click Manage in the Actions column to go to the instance details page.

Click the edit icon, enter a new instance name in the dialog box that appears, and then click Confirm.

Configure a cluster

On the details page of a new instance, the instance status is Pending Configuration. An empty cluster is automatically deployed based on the number and specifications of query nodes and data nodes that you purchased. Before you can perform searches, you must configure a data source, define an index, and then perform an index rebuild.

  1. Configure a data source. Supported data sources include MaxCompute and API. This example uses a MaxCompute data source. Click Add Data Source. Set Data Source Type to MaxCompute. Configure the project, accessKeyId, accessKeySecret, Table, and partition parameters. If required, enable automatic full indexing for the MaxCompute data source.

After the data source is verified, click OK to add it.

  1. After the data source is configured, click Next to configure the index schema.

2.1. Add an index table.

2.2. Configure the index table. Set Template to Vector: Text Semantic Search and Data Type to Convert raw data to vector data.

  • Index table: Customizable

  • Data Source: Select the data source that you configured in Step 1.

  • Data Shards: Configure this parameter based on the number of data nodes that you purchased.

2.3. Configure fields. The Vector: Text Semantic Search template automatically creates four preset fields: `id` (primary key), `vector_source_text` (the text field to be converted to a text embedding), `cate_id` (category field), and `vector` (the field that stores the text embedding). If you select a MaxCompute data source, the fields that are synchronized from the data source are displayed below the preset fields.

Advanced configuration for the vector field: Note: If you select Existing vector data, leave this configuration empty.

{
  "vector_model": "ops-text-embedding-000",
  "vector_modal": "text",
  "vector_source_field": "vector_source_text"
}

  • `vector_model`: The vector model. The following two models for Chinese and English conversion are supported:

    • `ops-text-embedding-000`: A model that converts short Chinese text to vectors. The dimension is fixed at 768.

    • `ops-text-embedding-en-000`: A model that converts short English text to vectors. The dimension is fixed at 768.

  • vector_modal: The vector type, which is text embedding.

  • `vector_source_field`: The field that contains the text to be vectorized. In this topic, this is `vector_source_text`.

Attribute and field content compression:

  • You can compress attribute fields. By default, they are not compressed. Select `file_compressor` to enable compression.

  • You can compress field content. By default, it is not compressed. For multi-value and STRING type fields, the default compression algorithm is `uniq`. For single-value numeric fields, the default is `equal`.

Note

  • When you use vector search, you must create fields in the following order: primary key field, tag field (optional), and vector field (as shown in the preceding figure).

  • The data in the `vector_source_text` field cannot exceed 128 bytes. If the data exceeds this limit, only the first 128 bytes are used for vector prediction.

  • If you use a MaxCompute data source, the fields that are synchronized from the data source are displayed below the preset fields.

  • The primary key field does not support compression.

  • Enabling field and attribute compression saves storage space but may decrease query performance. For more information, see the relevant documentation.

  • Field copying is supported. The new field (DUP field) has the same content as the original field. If different content is pushed to the DUP field, the content of the original field is used to overwrite it. If you want the field content to be different, you must manually delete the copy configuration in the advanced configuration of the DUP field.

Configure the index. The Vector: Text Semantic Search template automatically creates two preset indexes: a primary key index and a vector index. You can modify the preset indexes.

Set compression for index fields:

  • You can compress index fields. By default, they are not compressed. Select `file_compressor` to enable compression.

Note

  • The primary key index does not support compression.

  • Index compression saves storage space but may decrease query performance. For more information, see the relevant documentation.

3.1. Add included fields to the vector index:

Note

  • The primary key field and vector field are required. The tag field is optional and can be empty.

  • You can only select the three fixed fields. You cannot add new fields.

3.2. Advanced Configuration. You must configure the parameters for the vector index. You can use the following configuration as a reference. For more information, see Vector index.

More parameters:

The configuration for `build_index_params` is as follows:

{
  "proxima.qc.builder.quantizer_class": "Int8QuantizerConverter",
  "proxima.qc.builder.quantize_by_centroid": true,
  "proxima.qc.builder.optimizer_class": "BruteForceBuilder",
  "proxima.qc.builder.thread_count": 10,
  "proxima.qc.builder.optimizer_params": {
    "proxima.linear.builder.column_major_order": true
  },
  "proxima.qc.builder.store_original_features": false,
  "proxima.qc.builder.train_sample_count": 3000000,
  "proxima.qc.builder.train_sample_ratio": 0.5
}

The configuration for `search_index_params` is as follows:

{
  "proxima.qc.searcher.scan_ratio": 0.01
}

  1. After you complete the configuration, click Save Version. In the dialog box that appears, enter an optional note and click Publish.

After the index is published, click Next to rebuild the index.

  1. To rebuild the index, configure the parameters for the index rebuild and click Next.

  • API data source:

  • MaxCompute data source:

  1. Go to Operation Center > Change History > Data Source Changes to view the index rebuild progress. After the rebuild is complete, you can run a query test.

  1. Run a query test on the page.

Test the effect

Syntax introduction

query=text_index:'Text content&modal=text&n=10&search_params={}'

  • `modal` indicates the modality type. Set `modal` to text.

  • `n` specifies the number of top results to return from the vector search.

  • The text content must be Base64 encoded.

Retrieve data using an SDK

Add the dependency:

pip install alibabacloud-ha3engine

Search example:

# -*- coding: utf-8 -*-


from alibabacloud_ha3engine import models, client
from alibabacloud_tea_util import models as util_models
from Tea.exceptions import TeaException, RetryError
def search():
    Config = models.Config(
        endpoint="The API domain name under API Endpoint on the instance details page",
        instance_id="",
        protocol="http",
        access_user_name="The username set when purchasing the instance",
        access_pass_word="The password set when purchasing the instance"

    )

    # If a request takes a long time, use this configuration to increase the request wait time. Unit: ms.
    # This parameter can be used in the search_with_options method.
    runtime = util_models.RuntimeOptions(
        connect_timeout=5000,
        read_timeout=10000,
        autoretry=False,
        ignore_ssl=False,
        max_idle_conns=50
    )

    # Initialize the Ha3Engine client.
    ha3EngineClient = client.Client(Config)

    optionsHeaders = {}

    try:
        # Example 1: Search directly using an HA query string.
        # =====================================================
        query_str = "config=hit:4,format:json,fetch_summary_type:pk,qrs_chain:search&&query=text_index:'Text content&modal=text&n=10&search_params={}'&&cluster=general"
        haSearchQuery = models.SearchQuery(query=query_str)
        haSearchRequestModel = models.SearchRequestModel(optionsHeaders, haSearchQuery)
        hastrSearchResponseModel = ha3EngineClient.search(haSearchRequestModel)
        print(hastrSearchResponseModel)
    except TeaException as e:
        print(f"send request with TeaException : {e}")
    except RetryError as e:
        print(f"send request with Connection Exception  : {e}")

Notes

  • The vector index type must be set to **CUSTOMIZED**.

  • This scenario supports HA syntax and RESTful APIs but does not support SQL.

Previous：Image search quick startNext：Create and authorize RAM users