Prerequisites
Register an Alibaba Cloud account and complete identity verification.
You are prompted to create an AccessKey when you log on to the console with your Alibaba Cloud account for the first time.
An AccessKey is required for applications to make API requests. You must create an AccessKey for your Alibaba Cloud account.
After you create an AccessKey for your Alibaba Cloud account, you can also create one for a RAM user to access the service. Grant the RAM user the required permissions. For more information, see Authorization and authentication rules. Note: To access Vector Search Edition instances, the RAM user requires the `AliyunSearchEngineFullAccess` and `AliyunSearchEngineReadOnlyAccess` permissions.
3. A VPC environment is required. For more information, see this topic.
If you want to use a RAM user to manage Vector Search Edition instances, you must use your Alibaba Cloud account to grant the `AliyunSearchEngineFullAccess` and `AliyunSearchEngineReadOnlyAccess` permissions to the RAM user.
Purchase an instance
Log on to the OpenSearch console. In the upper-left corner, switch to OpenSearch-Vector Search Edition.
On the Instance Management page of the Vector Search Edition console, click Create Instance.
Set Product Version to Vector Search Edition and select a region. Configure Number of QRS Workers, QRS Worker Specifications, Number of Data Nodes, Data Node Specifications, and Total Storage per Data Node. Set the VPC and Virtual Switch. Set the username and password as prompted. These credentials are used for query authentication and are separate from your Alibaba Cloud account credentials. Click Buy Now.
Plan the number and specifications of QRS workers and data nodes based on your business needs. After you select the specifications, the cost is automatically calculated on the product page.
The VPC and virtual switch must be the same as the ones used by the ECS instance from which you will access the Vector Search Edition instance. Otherwise, an `{'errors':{'code':'403','message':'Forbidden'}}` error occurs.
Each data node includes a free storage quota. You can purchase additional storage, which is billed in 50 GB increments.
On the Confirm Order page, review the Terms of Service. Then, click Activate Now.
After the purchase is complete, click Management Console. Your new Vector Search Edition instance is displayed on the Instance Management page.
The new instance is created with a default name. To change the name, click Manage in the Actions column to go to the instance details page.
Click the edit icon, change the instance name as prompted, and then click Confirm.
Configure the cluster
A new instance has the Pending Configuration status on its details page. An empty cluster is automatically deployed with the QRS workers and data nodes that you purchased. To enable search, you must configure the cluster by performing the following steps in order: configure the data source, configure the index, and then rebuild the index.
Configure the data source. Supported data sources include MaxCompute data sources and data sources for data pushed over API. This topic uses a MaxCompute data source as an example. Click Add Data Source. Set Data Source Type to MaxCompute. Configure the Project, AccessKeyId, AccessKeySecret, Table, and Partition parameters. You can enable additional features for the MaxCompute data source as needed.
After the validation is successful, click OK to add the data source.
After the data source is configured, click Next to configure the index schema.
2.1. Add an index table.
2.2. Configure the index table. Select the General-purpose template.
Index Table: You can customize the name.
Data Source: Select the data source that you configured in Step 1.
Data Partitioning: Configure this parameter based on the number of data nodes that you purchased.
2.3. Define the fields. You must define at least two fields: a primary key field and a vector field. The vector field must be of a multi-value float type.
If you need vectors with categories, you can add a category field between the primary key and vector fields. The field type must be a single-value or multi-value integer.
Attribute and field content compression:
You can choose whether to compress attribute fields. By default, attribute fields are not compressed. To enable compression, select `file_compressor`.
You can choose whether to compress field content. By default, field content is not compressed. The default compression method for multi-value and STRING types is `uniq`. The default compression method for single-value numeric types is `equal`.
For vector search, you must define fields in the following order: primary key field, label field (optional), and vector field.
If you enable attribute compression, go to Deployment Management > Data Nodes > Online Table Configuration and edit the index loading method to reduce the impact on performance.
You can copy fields. The new field, also known as a DUP field, has the same content as the original field. If you push different content to the DUP field, the content of the original field is used instead. If you want the fields to have different content, you must manually delete the `copy from` setting in the advanced configuration of the DUP field.
Define the index. Set the index type for the primary key field to `PRIMARYKEY64`. For the vector index, select CUSTOMIZED as the index type.
Index field compression settings:
You can choose whether to compress index fields. By default, index fields are not compressed. To enable compression, select `file_compressor`.
Primary key indexes do not support compression.
If you enable index compression, go to Deployment Management > Data Nodes > Online Table Configuration and edit the index loading method to reduce the impact on performance.
3.1. Add included fields for the vector field.
The primary key field and vector field are required. The label field is optional.
Only these three fields can be selected. You cannot add other fields.
3.2. Configure advanced parameters. Vector indexes require separate parameter configuration. You can use the following configuration as a reference. For more information, see Vector indexes.
More parameters:
The configuration for `build_index_params` is as follows:
{
"proxima.qc.builder.quantizer_class": "Int8QuantizerConverter",
"proxima.qc.builder.quantize_by_centroid": true,
"proxima.qc.builder.optimizer_class": "BruteForceBuilder",
"proxima.qc.builder.thread_count": 10,
"proxima.qc.builder.optimizer_params": {
"proxima.linear.builder.column_major_order": true
},
"proxima.qc.builder.store_original_features": false,
"proxima.qc.builder.train_sample_count": 3000000,
"proxima.qc.builder.train_sample_ratio": 0.5
}The configuration for `search_index_params` is as follows:
{
"proxima.qc.searcher.scan_ratio": 0.01
}The system automatically populates the configuration parameters for the vector index. If you do not have special requirements, you can click OK to finish.
After the configuration is complete, click Save Version. In the pop-up dialog box, you can enter an optional comment. Then, click Publish.
After the index is published, click Next to rebuild the index.
Rebuild the index. Select the parameters for the rebuild and click Next.
Data source for data pushed over API:
MaxCompute data source:
Go to Operation Center > Change History > Data Source Changes to view the progress of the index rebuild. After the rebuild is complete, you can test your queries.
Test queries on the page.
Syntax description
General query:
HA3 syntax:
query=index_name:'0.1,0.2,0.98,0.6;0.3,0.4,0.98,0.6...'
Note: index_name is the name of the vector index, followed by the vector to query.SQL syntax:
query=select proxima_score('index_name') as score,id from table_name where MATCHINDEX('index_name', ?) order by score asc limit 5&&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;dynamic_params:[["0.892704,0.783731"]]
Note: index_name is the name of the vector index. `dynamic_params` in `kvpair` is the vector to query.Top-N query:
HA3 syntax:
query=index_name:'0.1,0.2,0.98,0.6;0.3,0.4,0.98,0.6&n=10'
Note: index_name is the name of the vector index, followed by the vector to query. n specifies the number of top results to return from the vector retrieval.SQL syntax:
query=select proxima_score('index_name') as score,id from table_name where MATCHINDEX('index_name', ?) order by score asc limit 5&&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;dynamic_params:[["0.892704,0.783731&n=10"]]
Note: index_name is the name of the vector index. `dynamic_params` in `kvpair` is the vector to query. n specifies the number of top results to return from the vector retrieval.Set a minimum score threshold:
HA3 syntax:
query=index_name:'0.1,0.2,0.98,0.6;0.3,0.4,0.98,0.6&n=10&sf=0.8'
Note: index_name is the name of the vector index, followed by the vector to query. sf specifies the threshold for filtering scores.SQL syntax:
query=select proxima_score('index_name') as score,id from table_name where MATCHINDEX('index_name', ?) order by score asc limit 5&&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;dynamic_params:[["0.892704,0.783731&n=10&sf=0.8"]]
Note: index_name is the name of the vector index. `dynamic_params` in `kvpair` is the vector to query. sf specifies the threshold for filtering scores.Set the retrieval parameters:
HA3 syntax:
query=index_name:'0.1,0.2,0.98,0.6;0.3,0.4,0.98,0.6&n=10&sf=0.8&search_params={"proxima.qc.searcher.scan_ratio":0.001,"proxima.general.searcher.scan_count":10000}'
Note: search_params specifies the vector retrieval parameters in JSON format. The meaning of proxima.qc.searcher.scan_ratio is the same as above. The meaning of proxima.general.searcher.scan_count is the same as min_scan_doc_cnt.
Note: The order of n, sf, and search_params cannot be changed.SQL syntax:
query=select proxima_score('index_name') as score,id from table_name where MATCHINDEX('index_name', ?) order by score asc limit 5&&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;dynamic_params:[["0.892704,0.783731&n=10&sf=0.8&search_params={"proxima.qc.searcher.scan_ratio":0.001,"proxima.general.searcher.scan_count":10000}"]]
Note: index_name is the name of the vector index. `dynamic_params` in `kvpair` is the vector to query. search_params specifies the vector retrieval parameters in JSON format. The meaning of proxima.qc.searcher.scan_ratio is the same as above. The meaning of proxima.general.searcher.scan_count is the same as min_scan_doc_cnt.
Note: The order of n, sf, and search_params cannot be changed.If a category field is configured in the vector index, you must include it in the retrieval query for vector searches. Otherwise, data cannot be retrieved correctly. For example:
query=select proxima_score('vector_index_name') as score, cate_id from table_name where MATCHINDEX('vector_index_name', ?) &&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;trace:INFO;formatType:json;dynamic_params:[["168#0.01747940666973591,-0.03125246614217758,-0.03254068270325661..."]]In `dynamic_params`, `168#` means that `168` is the value of the category field, followed by the `#` separator.
Use vector search in an SDK
Add the dependency:
pip install alibabacloud-ha3engineSearch example:
from alibabacloud_ha3engine import models, client
from alibabacloud_tea_util import models as util_models
from Tea.exceptions import TeaException, RetryError
def search():
Config = models.Config(
endpoint="",
instance_id="",
protocol="http",
access_user_name="",
access_pass_word=""
)
# If the user request takes a long time, you can use this configuration to increase the request timeout period. Unit: ms.
# This parameter can be used in the search_with_options method.
runtime = util_models.RuntimeOptions(
connect_timeout=5000,
read_timeout=10000,
autoretry=False,
ignore_ssl=False,
max_idle_conns=50
)
# Initialize the Ha3Engine client.
ha3EngineClient = client.Client(Config)
optionsHeaders = {}
try:
# Example 3: Use an HA-SQL query string to search directly.
# =====================================================
sql_str = '''query=select proxima_score('index_name') as score,id from table_name where MATCHINDEX('index_name', ?) order by score asc limit 5&&kvpair=timeout:1000,iquan.plan.cache.enable:true;urlencode_data:false;iquan.plan.prepare.level:jni.post.optimize;dynamic_params:[["0.892704,0.783731&n=10"]]'''
sqlsearchQuery = models.SearchQuery(sql=sql_str)
sqlSearchRequestModel = models.SearchRequestModel(optionsHeaders, sqlsearchQuery)
sqlstrSearchResponseModel = ha3EngineClient.search(sqlSearchRequestModel)
print(sqlstrSearchResponseModel)
except TeaException as e:
print(f"send request with TeaException : {e}")
except RetryError as e:
print(f"send request with Connection Exception : {e}")
if(__name__ == "__main__"):
search()