In multi-tenant scenarios with large-scale indexes, custom routing keys are a core technology for user-level data isolation and precise queries. By binding a user identity, such as a user ID, to a routing key, you can ensure that each query targets only a specific user's data. This improves both data security and query performance. This topic describes how to use the custom routing key feature.
Prerequisites
The Lindorm vector engine is enabled. For more information, see Enable the vector engine.
The Lindorm search engine is enabled. For more information, see Activation guide.
The IP address of the client is added to the whitelist of the Lindorm instance. For more information, see Configure a whitelist.
Preparations
Before you can use advanced features, you must connect to the search engine using the curl command. For more information about the connection method and parameters, see Connect to the search engine.
Create an index
For pure vector data queries only
If the data volume in the index is less than 10,000, use a flat index. If the data volume is in the tens or hundreds of thousands, use an hnsw index. If the data volume reaches millions, use an ivfpq index. You can also use a sparse vector index as needed.
In custom routing key scenarios, the primary key `_id` must be globally unique.
When you create the index, specify
"knn_routing": trueto enable the custom routing key feature. For an ivfpq index, also set"meta": {"offline.construction": "true"}.
flat routing index
curl -u <username>:<password> -H 'Content-Type: application/json' -XPUT "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/vector_routing_flat_test?pretty" -d '
{
"settings" : {
"index": {
"number_of_shards": 2,
"knn": true,
"knn_routing": true
}
},
"mappings": {
"_source": {
"excludes": ["vector1"]
},
"properties": {
"vector1": {
"type": "knn_vector",
"dimension": 3,
"data_type": "float",
"method": {
"engine": "lvector",
"name": "flat",
"space_type": "l2",
"parameters": {}
}
},
"field1": {
"type": "long"
}
}
}
}
'hnsw routing index
curl -u <username>:<password> -H 'Content-Type: application/json' -XPUT "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/vector_routing_hnsw_test?pretty" -d '
{
"settings" : {
"index": {
"number_of_shards": 2,
"knn": true,
"knn_routing": true
}
},
"mappings": {
"_source": {
"excludes": ["vector1"]
},
"properties": {
"vector1": {
"type": "knn_vector",
"dimension": 3,
"method": {
"engine": "lvector",
"name": "hnsw",
"space_type": "l2",
"parameters": {
"m": 24,
"ef_construction": 500
}
}
},
"field1": {
"type": "long"
}
}
}
}'sparse_hnsw routing sparse vector index
curl -u <username>:<password> -H 'Content-Type: application/json' -XPUT "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/vector_routing_sparse_test?pretty" -d '
{
"settings" : {
"index": {
"number_of_shards": 2,
"knn": true,
"knn_routing": true
}
},
"mappings": {
"_source": {
"excludes": ["vector1"]
},
"properties": {
"vector1": {
"type": "knn_vector",
"data_type": "sparse_vector",
"method": {
"engine": "lvector",
"name": "sparse_hnsw",
"space_type": "innerproduct",
"parameters": {
"m": 24,
"ef_construction": 200
}
}
},
"field1": {
"type": "long"
}
}
}
}'ivfpq routing index
In custom routing key scenarios, the data volume for a single routing key is usually small, such as hundreds of thousands of records or fewer. The ivfpq parameter settings for these scenarios differ from the general policies for data volumes in the tens or hundreds of millions. For example, when you set the nlist parameter, which defines the number of clusters, follow the principle that each cluster should contain 1,000 to 30,000 records. If the data volume for each routing key is a few thousand records, set nlist to 2.
curl -u <username>:<password> -H 'Content-Type: application/json' -XPUT "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/vector_routing_ivfpq_test?pretty" -d '
{
"settings": {
"index": {
"number_of_shards": 4,
"knn": true,
"knn_routing": true
}
},
"mappings": {
"_source": {
"excludes": ["vector1"]
},
"properties": {
"vector1": {
"type": "knn_vector",
"dimension": 3,
"data_type": "float",
"meta": {"offline.construction": "true"},
"method": {
"engine": "lvector",
"name": "ivfpq",
"space_type": "cosinesimil",
"parameters": {
"m": 3, // Set to the same value as the dimension.
"nlist": 2,
"centroids_use_hnsw": false,
"centroids_hnsw_m": 48,
"centroids_hnsw_ef_construct": 500,
"centroids_hnsw_ef_search": 200
}
}
},
"field1": {
"type": "long"
}
}
}
}
'Support for pure vector data and hybrid queries
To perform a hybrid query, specify a full-text index field when you create the index. To do this, add the following parameter:
"text_field": {
"type": "text",
"analyzer": "ik_max_word"
}The following statement provides an example of how to create an hnsw routing index:
curl -u <username>:<password> -H 'Content-Type: application/json' -XPUT "http://ld-bp1h002998iv8****-proxy-search-pub.lindorm.aliyuncs.com:30070/vector1_routing_hnsw_hybirdSearch?pretty" -d '
{
"settings" : {
"index": {
"number_of_shards": 2,
"knn": true,
"knn_routing": true
}
},
"mappings": {
"_source": {
"excludes": ["vector1"]
},
"properties": {
"vector1": {
"type": "knn_vector",
"dimension": 3,
"data_type": "float",
"method": {
"engine": "lvector",
"name": "hnsw",
"space_type": "l2",
"parameters": {
"m": 24,
"ef_construction": 500
}
}
},
"text_field": {
"type": "text",
"analyzer": "ik_max_word"
},
"field1": {
"type": "long"
}
}
}
}'Write data
Write a single record
The following example shows how to write data to the flat index vector_routing_flat_test and specify the routing value for the tenant user123.
curl -u <username>:<password> -H 'Content-Type: application/json' -XPUT "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/vector_routing_flat_test/_doc/1?routing=user123" -d '
{
"vector1": [1.2, 1.3, 1.4],
"field1": 1
}
'Write records in batches
The following example shows how to write data in batches to the hnsw index vector_routing_hnsw_test and specifies the routing values 1 and 2.
curl -u <username>:<password> -H "Content-Type: application/json" -XPOST "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/_bulk?pretty" -d '
{ "index" : { "_index" : "vector_routing_hnsw_test", "_id" : "2", "routing": "1"} }
{ "field1" : 2, "vector1": [2.2, 2.3, 2.4]}
{ "index" : { "_index" : "vector_routing_hnsw_test", "_id" : "3", "routing": "2" } }
{ "field1" : 3, "vector1": [3.2, 3.3, 3.4]}
'Write sparse vectors in batches
curl -u <username>:<password> -H "Content-Type: application/json" -XPOST "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/_bulk?pretty" -d '
{ "index" : { "_index" : "vector_routing_sparse_test", "_id" : "2", "routing": "1"} }
{ "field1" : 2, "vector1": {"indices": [10, 12, 16], "values": [0.5, 0.5, 0.2]}}
{ "index" : { "_index" : "vector_routing_sparse_test", "_id" : "3", "routing": "2" } }
{ "field1" : 3, "vector1": {"indices": [10, 12, 16], "values": [0.5, 0.5, 0.2]}}
'Build an index
Build an ivfpq index
Only ivfpq indexes must be built manually. In the build statement, you must set "meta": {"offline.construction": "true"}. This setting indicates an offline index.
Before you start the build, ensure that a sufficient amount of data has been written to the index. The number of records must be greater than 256 and more than 30 times the value of nlist.
curl -u <username>:<password> -H 'Content-Type: application/json' -XPOST "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/_plugins/_vector/index/build" -d '
{
"indexName": "vector_routing_ivfpq_test",
"fieldName": "vector1",
"removeOldIndex": "true",
"ivf_train_only": "false"
}'Parameter description
Parameter | Required | Description |
ivf_train_only | Yes |
Regardless of whether this parameter is set to |
Clear training data and retain the index codebook
You must perform this step only if you set ivf_train_only to true. This operation uses the existing data to train the codebook but does not generate an index for the existing data.
The reserve_codebook=true parameter is required. This parameter specifies that the index codebook is saved. After you clear the training data, you must write the data again before you can perform pure vector data queries (k-NN searches).
If you set ivf_train_only to false, an index is generated for the existing data based on the trained codebook, and the existing data is retained. You can skip this step.
curl -u <username>:<password> -H "Content-Type: application/json" -XPOST "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/_truncate/vector_routing_ivfpq_test?pretty&reserve_codebook=true"Query data
Query pure vector data
You can query pure vector data using the knn structure.
flat routing index
curl -u <username>:<password> -H "Content-Type: application/json" -XPOST "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/vector_routing_flat_test/_search?pretty&routing=1" -d '{
"size": 20,
"query": {
"knn": {
"vector1": {
"vector": [2.3, 3.3, 4.4],
"k": 20
}
}
}
}'hnsw routing index
curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/vector_routing_hnsw_test/_search?pretty&routing=1" -d '
{
"size": 10,
"query": {
"knn": {
"vector1": {
"vector": [2.2, 2.3, 2.4],
"k": 10
}
}
},
"ext": {"lvector": {"ef_search": "100"}}
}'sparse_hsnw routing sparse vector index
curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/vector_routing_sparse_test/_search?pretty&routing=1" -d '
{
"size": 10,
"query": {
"knn": {
"vector1": {
"vector": {"indices": [10, 45, 16], "values": [0.5, 0.5, 0.2]},
"k": 10
}
}
},
"ext": {"lvector": {"ef_search": "100"}}
}'ivfpq routing index
curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://ld-bp1h002998iv8****.lindorm.aliyuncs.com:30070/vector_ivfpq_test/_search?pretty&routing=1" -d '
{
"size": 10,
"query": {
"knn": {
"vector1": {
"vector": [2.2, 2.3, 2.4],
"k": 10
}
}
},
"ext": {"lvector": {"nprobe": "2", "reorder_factor": "2","client_refactor":"true"}}
}'In custom routing key scenarios, you can set the value of nprobe to the value of the nlist parameter that you specified when you created the index.
Perform a hybrid query
Before you perform a hybrid query, ensure that you have specified a full-text index field for your index.
Hybrid full-text and vector search
curl -u <username>:<password> -H "Content-Type: application/json" -XPOST "http://ld-bp1h002998iv8****-proxy-search-vpc.lindorm.rds.aliyuncs.com:30070/vector_text_hybridSearch/_search?pretty&routing=1" -d '{
"size": 10,
"_source": false,
"query": {
"knn": {
"vector1": {
"vector": [2.8, 2.3, 2.4],
"filter": {
"bool": {
"must": [{
"bool": {
"must": [{
"match": {
"text_field": { // Replace with the full-text field that you want to search.
"query": "test1 test2"
}
}
},
{
"term": {
"_routing": "Replace with the routing value specified in the request URL, such as 1 or user123"
}
}]
}
}]
}
},
"k": 10
}
}
},
"ext": {
"lvector": {
"hybrid_search_type": "filter_rrf",
"rrf_rank_constant": "60",
"rrf_knn_weight_factor": "0.5"
}
}
}'Vector, full-text, and property filtering
curl -u <username>:<password> -H "Content-Type: application/json" -XPOST "http://ld-bp1h002998iv8****-proxy-search-vpc.lindorm.rds.aliyuncs.com:30070/vector_text_hybridSearch/_search?pretty&routing=1" -d '{
"size": 10,
"_source": false,
"query": {
"knn": {
"vector1": {
"vector": [2.8, 2.3, 2.4],
"filter": {
"bool": {
"must": [{
"bool": {
"must": [{
"match": {
"text_field": { // Replace with the full-text field that you want to search.
"query": "test1 test2"
}
}
},
{
"term": {
"_routing": "Replace with the routing value specified in the request URL, such as 1 or user123"
}
}]
}
},
{
"bool": {
"filter": [{
"range": {
"field1": {
"gt": 2
}
}
}]
}
}]
}
},
"k": 10
}
}
},
"ext": {
"lvector": {
"hybrid_search_type": "filter_rrf",
"rrf_rank_constant": "60",
"rrf_knn_weight_factor": "0.5",
"filter_type": "efficient_filter"
}
}
}'