Vector search

Prerequisites

Install the Tablestore SDK for Java and initialize the client. Vector search requires version 5.17.0 or later.
To configure minScore or numCandidates, use version 5.17.5 or later.

Feature description

Vector search performs an approximate nearest neighbor (ANN) search between a query vector and vectors in a Vector field. Tablestore scores the results with the distance metric configured when the search index was created and returns the nearest data. Unlike queries that match field values, vector search determines similarity from the distance between vectors.

Call search and set query to KnnVectorQuery.

SearchResponse search(SearchRequest request)

The following example retrieves the three vectors in the embedding field that are nearest to [1.0, 0.0, 0.0, 0.0]. The results are sorted by score in descending order.

String tableName = "example_table";
String indexName = "example_index";
KnnVectorQuery query = new KnnVectorQuery();
query.setFieldName("embedding");
query.setTopK(3);
query.setFloat32QueryVector(new float[]{1.0f, 0.0f, 0.0f, 0.0f});

SearchQuery searchQuery = new SearchQuery();
searchQuery.setQuery(query);
searchQuery.setLimit(3);
searchQuery.setSort(new Sort(Collections.singletonList(new ScoreSort())));

SearchRequest request = new SearchRequest(tableName, indexName, searchQuery);
SearchRequest.ColumnsToGet columnsToGet = new SearchRequest.ColumnsToGet();
columnsToGet.setReturnAll(true);
request.setColumnsToGet(columnsToGet);

SearchResponse response = client.search(request);
for (SearchHit hit : response.getSearchHits()) {
    System.out.println(hit.getScore() + ": " + hit.getRow());
}

Note

Limits apply to the number and dimensions of vector fields and to topK. For details, see Search index limits.

Parameters

Search request

request is a SearchRequest object that contains the following parameters.

Name	Type	Description
tableName (required)	String	The table name.
indexName (required)	String	The search index name.
searchQuery (required)	SearchQuery	The query condition and common query configurations.
columnsToGet (optional)	SearchRequest.ColumnsToGet	The returned column configurations. If this parameter is not configured, only primary key columns are returned.
timeoutInMillisecond (optional)	int	The request-level query timeout in milliseconds. Default: `-1`, which does not configure a separate query timeout.
routingValues (optional)	`List<PrimaryKey>`	The primary key values of custom routing fields. Leave this parameter unset if custom routing is not configured.

Query configuration

request.searchQuery is a SearchQuery object that contains the following parameters.

Name	Type	Description
query (required)	Query	The query condition. Set this parameter to `KnnVectorQuery` for vector search.
offset (optional)	Integer	The position from which the query starts.
limit (optional)	Integer	The maximum number of rows to return. If this parameter is set to `0`, no rows are returned.
highlight (optional)	Highlight	The summary and highlighting configurations. Vector fields do not support summary and highlighting.
collapse (optional)	Collapse	The collapse configuration for deduplicating results based on a specified column.
sort (optional)	Sort	The result sort order. Use `ScoreSort` to sort by score.
trackTotalCount (optional)	int	The maximum number of matching rows to count. Default: `TRACK_TOTAL_COUNT_DISABLED`, which disables counting. Set this parameter to `TRACK_TOTAL_COUNT` to count all matching rows. A smaller value improves query performance.
filter (optional)	SearchFilter	A filter applied to the results of `query`.
aggregationList (optional)	`List<Aggregation>`	The aggregation configurations.
groupByList (optional)	`List<GroupBy>`	The grouping configurations.
token (optional)	byte[]	The pagination token. Set this parameter to the `nextToken` from the previous response to retrieve more data. Each server-side index partition returns its own `topK` nearest values, which are merged at a coordinating node. Therefore, when you paginate with `token`, the cumulative number of returned rows depends on the number of server-side index partitions.

Vector query condition

request.searchQuery.query is a KnnVectorQuery object that contains the following parameters.

Name	Type	Description
fieldName (required)	String	The vector field name. The field must be of the Vector type, and the query vector dimension must match the dimension configured when you created the search index.
topK (required)	Integer	The number of nearest vectors to retrieve. Maximum: `1000`. A larger value returns more candidates and may improve recall, but can also increase query latency and cost.
float32QueryVector (required)	float[]	The Float32 query vector used to calculate similarity. The array length must match the vector field dimension.
filter (optional)	Query	Non-vector query conditions that vector search results must also meet. You can combine multiple non-vector `Query` objects.
weight (optional)	Float	The relevance weight of the vector query. The value must be greater than or equal to `0`. Default: `1.0`. A larger value gives the vector query score more influence on the final relevance score without changing which rows match.
minScore (optional)	Float	The minimum score threshold. The value must be greater than or equal to `0`. Default: `0`. Only data with a score strictly greater than this value is returned.
numCandidates (optional)	Integer	The number of candidates accessed in each index partition when nearest neighbors are calculated. Valid values: `[topK, 1000]`. A larger value may improve recall but can also increase query time.

Returned columns

request.columnsToGet is a SearchRequest.ColumnsToGet object that contains the following parameters.

Name	Type	Description
columns (optional)	`List<String>`	The attribute columns to return. Configure this parameter only if both `returnAll` and `returnAllFromIndex` are `false`. If this parameter is not configured, only primary key columns are returned.
returnAll (optional)	boolean	Specifies whether to return all attribute columns in the table. Default: `false`.
returnAllFromIndex (optional)	boolean	Specifies whether to return all indexed attribute columns. Default: `false`. This parameter and `returnAll` cannot both be set to `true`.

Response

Search response

search returns a SearchResponse object. The following table describes the core fields.

Name	Type	Description
totalCount	long	The number of matching rows. Call `getTotalCount()` to obtain the value. The returned value depends on `trackTotalCount`.
rows	`List<Row>`	The rows returned by this query. Call `getRows()` to obtain the value. The number of returned rows does not exceed `limit`.
searchHits	`List<SearchHit>`	The query hits. Call `getSearchHits()` to obtain the rows and their scores.
nextToken	byte[]	The token for the next page. Call `getNextToken()` to obtain the value. If the value is not `null`, pass it as `token` in the next request to retrieve more data.
isAllSuccess	boolean	Indicates whether all index partitions were queried. Call `isAllSuccess()` to obtain the value. If this field is `false`, the response contains partial results and `totalCount` may be less than the actual number of matching rows.

Search hit

Each element in response.searchHits[] is a SearchHit object that contains the following core fields.

Name	Type	Description
row	Row	The matching row. Call `getRow()` to obtain the value.
score	Double	The vector query score. Call `getScore()` to obtain the value. The distance metric and `weight` affect this value.

Examples

Filter by non-vector conditions and a minimum score

Use filter to require nearest neighbors to meet non-vector query conditions, and use minScore to exclude data whose score is not greater than the threshold. The following example returns only data where category is book, price is less than 4, and the vector score is greater than 0.6.

KnnVectorQuery filteredQuery = new KnnVectorQuery();
filteredQuery.setFieldName("embedding");
filteredQuery.setTopK(10);
filteredQuery.setFloat32QueryVector(
        new float[]{1.0f, 0.0f, 0.0f, 0.0f});
filteredQuery.setMinScore(0.6f);
filteredQuery.setFilter(QueryBuilders.bool()
        .must(QueryBuilders.term("category", "book"))
        .must(QueryBuilders.range("price").lessThan(4)));

SearchQuery searchQuery = new SearchQuery();
searchQuery.setQuery(filteredQuery);
searchQuery.setLimit(10);

Adjust the candidate count

Set numCandidates to expand the candidate set accessed in each index partition when nearest neighbors are calculated. The following example retrieves the three nearest vectors from four candidates.

KnnVectorQuery candidateQuery = new KnnVectorQuery();
candidateQuery.setFieldName("embedding");
candidateQuery.setTopK(3);
candidateQuery.setFloat32QueryVector(
        new float[]{1.0f, 0.0f, 0.0f, 0.0f});
candidateQuery.setNumCandidates(4);

SearchQuery candidateSearchQuery = new SearchQuery();
candidateSearchQuery.setQuery(candidateQuery);
candidateSearchQuery.setLimit(3);