Basic features

Insert

The operation fails if a document with the same ID already exists.

curl -u <username>:<password> -H 'Content-Type: application/json' -XPOST "http://<URL>/_bulk?pretty"  -d '
{ "create" : { "_index" : "vector_test", "_id" : "1" } }
{ "field1" : 1, "vector1": [2.2, 2.3, 2.4]}
{ "create" : { "_index" : "vector_test", "_id" : "2" } }
{ "field1" : 2, "vector1": [1.2, 1.3, 4.4]}
'

Overwrite

If a document with the specified ID does not exist, a new one is created. If a document with the same ID already exists, the existing document is overwritten. For example, if you write a document with only a vector field, and a subsequent write for the same ID contains only a scalar field, the original vector field is deleted.

curl -u <username>:<password> -H 'Content-Type: application/json' -XPOST "http://<URL>/_bulk?pretty"  -d '
{ "index" : { "_index" : "vector_test", "_id" : "3" } }
{ "field1" : 1, "vector1": [2.2, 2.3, 2.4]}
{ "index" : { "_index" : "vector_test", "_id" : "4" } }
{ "field1" : 2, "vector1": [1.2, 1.3, 4.4]}
'

Sparse vector write

The write method is the same as previously described, but the format of the vector1 field is different. Two formats are supported: JSON STRING and JSON Object. The JSON STRING format provides better performance, while the JSON Object format is more user-friendly.

curl -u <username>:<password> -H 'Content-Type: application/json' -XPUT "http://<URL>/vector_sparse_test/_doc/1?pretty"  -d '{
  "field1": 2, 
  "vector1": "{\"indices\": [10, 14, 16], \"values\": [0.5, 0.5, 0.2]}"
}'

curl -u <username>:<password> -H 'Content-Type: application/json' -XPUT "http://<URL>/vector_sparse_test/_doc/1?pretty"  -d '{
  "field1": 1, 
  "vector1": {"indices": [10, 12, 16], "values": [0.5, 0.5, 0.2]}
}'

Parameters

Return specific fields

To return only specific fields in the query results, specify "_source": ["field1", "field2"] . To return all non-vector fields, use "_source": true. The following example shows how to query the vector_test index:

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://<URL>/vector_test/_search?pretty"  -d '
{
  "size": 10,
  "_source": ["field1"],
  "query": {
    "knn": {
      "vector1": {
        "vector": [2.2, 2.3, 2.4],
        "k": 10
      }
    }
  },
  "ext": {"lvector": {"min_score": "0.01"}}
}'

The query returns the following response:

{
  "took" : 35,
  "timed_out" : false,
  "terminated_early" : false,
  "num_reduce_phases" : 0,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "vector_test",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "field1" : 1
        }
      },
      {
        "_index" : "vector_test",
        "_id" : "1",
        "_score" : 0.25,
        "_source" : {
          "field1" : 1
        }
      },
      {
        "_index" : "vector_test",
        "_id" : "3",
        "_score" : 0.14285715,
        "_source" : {
          "field1" : 2
        }
      }
    ]
  }
}

HNSW and HNSWQ query

The following example shows how to query the HNSW index vector_test:

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://<URL>/vector_test/_search?pretty"  -d '
{
  "size": 10,
  "query": {
    "knn": {
      "vector1": {
        "vector": [2.2, 2.3, 2.4],
        "k": 10
      }
    }
  },
  "ext": {"lvector": {"ef_search": "100"}}
}'

IVFPQ and IVFBQ query

The IVFPQ and IVFBQ algorithms use the same query parameters. The following example shows how to query the IVFPQ index vector_ivfpq_test:

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://<URL>/vector_ivfpq_test/_search?pretty"  -d '
{
  "size": 10,
  "query": {
    "knn": {
      "vector1": {
        "vector": [2.2, 2.3, 2.4],
        "k": 10
      }
    }
  },
  "ext": {"lvector": {"nprobe": "60", "reorder_factor": "2"}}
}'

Sparse vector query

The query method is the same, but the vector1 field format differs. Sparse vectors support two formats: JSON String for better performance and JSON Object for user-friendliness.

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://<URL>/vector_sparse_test/_search?pretty"  -d '
{
  "size": 10,
  "query": {
    "knn": {
      "vector1": {
        "vector": "{\"indices\": [10, 45, 16], \"values\": [0.5, 0.5, 0.2]}",
        "k": 10
      }
    }
  }
}'

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://<URL>/vector_sparse_test/_search?pretty"  -d '
{
  "size": 10,
  "query": {
    "knn": {
      "vector1": {
        "vector": {"indices": [10, 45, 16], "values": [0.5, 0.5, 0.2]},
        "k": 10
      }
    }
  }
}'

Sparse vector inverted index query

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://ld-t631ui****.lindorm.aliyuncs.com:30070/vector_sparse_inverted_test/_search?pretty"  -d '
{
  "size": 10,
  "query": {
    "knn": {
      "vector1": {
        "vector": "{\"indices\": [10, 45, 16], \"values\": [0.5, 0.5, 0.2]}",
        "k": 10
      }
    }
  },
  "ext": {"lvector": {"search_algo": "max_score", "threshold_factor":"1.5"}}
}'

The following are the ext.lvector parameters for sparse vector inverted index queries. For information about other parameters, see Parameter descriptions.

Pre-filtering approximate search

To filter structured data before the vector search, add a filter clause to the knn query and set the filter_type parameter to pre_filter.

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://<URL>/<index_name>/_search?pretty"  -d '
{
  "size": 10,
  "query": {
    "knn": {
      "vector1": {
        "vector": [2.3, 3.3, 4.4],
        "filter": {
          "range": {
            "field1": {
              "gte": 0
            }
          }
        },
        "k": 10
      }
    }
  },
  "ext": {"lvector": {"filter_type": "pre_filter"}}
}'

Post-filtering approximate search

To filter results after the vector search, add a filter clause to the knn query and set the filter_type parameter to post_filter.

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://<URL>/<index_name>/_search?pretty"  -d '
{
  "size": 10,
  "query": {
    "knn": {
      "vector1": {
        "vector": [2.3, 3.3, 4.4],
        "filter": {
          "range": {
            "field1": {
              "gte": 0
            }
          }
        },
        "k": 10
      }
    }
  },
  "ext": {"lvector": {"filter_type": "post_filter"}}
}'

When you use post-filtering for an approximate search, you must increase k. If you use the IVFPQ algorithm, you must also adjust reorder_factor. For example:

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://<URL>/<ivfpq_index_name>/_search?pretty"  -d '
{
  "size": 10,
  "query": {
    "knn": {
      "vector1": {
        "vector": [2.2, 2.3, 2.4],
        "filter": {
          "range": {
            "field1": {
              "gte": 0
            }
          }
        },
        "k": 1000
      }
    }
  },
  "ext": {"lvector": {"filter_type": "post_filter","nprobe": "60", "reorder_factor": "1"}}
}'

You can also add filter conditions using the post_filter clause to perform an approximate search with post-filtering.

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://<URL>/<index_name>/_search?pretty"  -d '
{
  "size": 10,
  "query": {
    "knn": {
      "vector1": {
        "vector": [2.3, 3.3, 4.4],
        "k": 10
      }
    }
  },
  "post_filter": {
    "range": {
      "field1": {
        "gte": 0
      }
    }
  }
}'

Efficient filtering approximate search

For an approximate search, you can explicitly choose a filtering strategy by setting the filter_type parameter to pre_filter or post_filter. To let the system choose automatically, set the filter_type parameter to efficient_filter, which uses an internal cost estimation to select either the pre_filter or post_filter strategy.

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://<URL>/<index_name>/_search?pretty"  -d '
{
  "size": 10,
  "query": {
    "knn": {
      "vector1": {
        "vector": [2.2, 2.3, 2.4],
        "filter": {
          "range": {
            "field1": {
              "gte": 0
            }
          }
        },
        "k": 10
      }
    }
  },
  "ext": {"lvector": {"filter_type": "efficient_filter", "ef_search":"
100", "k_expand_scope": "2000"}}
}'

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://<URL>/<index_name>/_search?pretty"  -d '
{
  "size": 10,
  "query": {
    "knn": {
      "vector1": {
        "vector": [2.2, 2.3, 2.4],
        "filter": {
          "range": {
            "field1": {
              "gte": 0
            }
          }
        },
        "k": 10
      }
    }
  },
  "ext": {"lvector": {"filter_type": "efficient_filter","nprobe": "80", "reorder_factor": "2","client_refactor":"true", "k_expand_scope": "2000"}}
}'

Attribute filter syntax

The filter clause adds filter conditions and accepts standard OpenSearch query DSL.

"filter": {
  "range": {
    "field1": {
      "gte": 0
    }
  }
}

"filter": {
  "bool": {
    "should": [{
      "range": {
        "field1": {
          "gt": 2
        }
      }
    },
    {
      "term": {
        "field2": "flag2"
      }
    }]
  }
}