When querying data from a search index, you can sort the results by predefining an order or specifying one at query time. For large result sets, use offset-based or token-based paging to quickly find the data you need.
Use cases
Category | Method | Feature | Use case |
Sorting | Define at creation time | By default, Tablestore sorts data in a search index using the configured index presorting ( | |
Define at query time | Sorts query results by relevance score, which is calculated using the BM25 algorithm. This is suitable for relevance-based scenarios, such as full-text search. | ||
Sorts results by primary key. This is useful for sorting items by their unique identifiers. | |||
Sorts results by field value. This is useful for e-commerce or social media scenarios where you might sort by attributes such as sales volume or page views. For multi-valued fields, such as array type or nested type fields, you can use the | |||
Sorts results by the distance from a geographic point. This is suitable for location-based services such as maps or logistics, for example, sorting nearby restaurants by distance. | |||
Paging | Define at query time | Use offset-based paging when the total number of rows to retrieve is less than 100,000. | |
Use token-based paging to sequentially retrieve large result sets. By default, you can only page forward. However, because a token remains valid throughout the query process, you can cache a previous token to page backward. |
SDKs
You can implement sorting and paging using the SDKs for the following languages.
Index presorting
By default, Tablestore sorts data in a search index based on a predefined sort order, called index presorting (IndexSort). When you query data, IndexSort determines the default order of the returned results.
When you create a search index, you can customize IndexSort. If you do not specify this setting, the index defaults to primary key sort.
Index presorting supports only
PrimaryKeySort(primary key sort) andFieldSort(field sort).You cannot use index presorting for a search index that contains nested type fields.
After you create a search index, the dynamic schema modification feature enables you to modify the
IndexSortsettings.
Sort at query time
You can sort only on fields where the enableSortAndAgg property is set to true.
You can specify a sorting method for each query. A search index supports the following four types of sorters. You can also use multiple sorters to sort results based on a sequence of criteria.
ScoreSort
Sorts query results by relevance score, which is calculated using the BM25 algorithm. This method is suitable for relevance-based scenarios, such as full-text search.
If you want to sort by relevance score, you must explicitly specify
ScoreSort. Otherwise, Tablestore sorts the results according to theIndexSortsettings of the index.When you use
ScoreSort, fields of the FuzzyKeyword type are not included in the sorting, and theweightparameter has no effect on these fields.
SearchQuery searchQuery = new SearchQuery();
searchQuery.setSort(new Sort(Arrays.asList(new ScoreSort())));PrimaryKeySort
Sorts results by primary key.
SearchQuery searchQuery = new SearchQuery();
searchQuery.setSort(new Sort(Arrays.asList(new PrimaryKeySort()))); // In ascending order.
//searchQuery.setSort(new Sort(Arrays.asList(new PrimaryKeySort(SortOrder.DESC)))); // In descending order.FieldSort
Sorts results by field value.
Single-field sort
Sorts results based on the values in a single field.
SearchQuery searchQuery = new SearchQuery();
searchQuery.setSort(new Sort(Arrays.asList(new FieldSort("col", SortOrder.ASC))));Multi-field sort
Sorts results first by the values in one field, and then by the values in another field.
SearchQuery searchQuery = new SearchQuery();
searchQuery.setSort(new Sort(Arrays.asList(
new FieldSort("col1", SortOrder.ASC), new FieldSort("col2", SortOrder.ASC))));Fallback field
When you sort by a field of the Long, Double, or Date type, you can set the missingField parameter. This parameter specifies another field of the same type to use as a fallback value for sorting if a row is missing a value in the primary sort field.
/**
* Sort results in descending order based on the values in the Col_Long field.
* If a row is missing a value in the Col_Long field (Long type),
* the value from the Col_Long_sec field (Long type) is used for sorting instead.
*/
SearchQuery searchQuery = new SearchQuery();
FieldSort fieldSort = new FieldSort("Col_Long");
// Specify the Col_Long_sec field as the fallback for sorting when a value is missing in the Col_Long field.
fieldSort.setMissingField("Col_Long_sec");
fieldSort.setOrder(SortOrder.DESC); Missing values
The missingValue parameter specifies the sort position for documents that are missing the sort field. You can set this parameter to control where these documents appear in the results.
The sorting behavior is as follows:
If you set
missingValuetoFieldSort.FIRST_WHEN_MISSING, documents with missing values are always placed at the beginning of the results, regardless of the sort order (ascending or descending).If you set
missingValuetoFieldSort.LAST_WHEN_MISSINGornull, documents with missing values are always placed at the end of the results, regardless of the sort order./** * Sort results in descending order based on the values in the Col_Long field (Long type). * If a row is missing a value in the Col_Long field, place that document at the beginning of the results. */ SearchQuery searchQuery = new SearchQuery(); FieldSort fieldSort = new FieldSort("Col_Long"); // Place documents with missing values first. fieldSort.setMissingValue(FieldSort.FIRST_WHEN_MISSING); fieldSort.setOrder(SortOrder.DESC); searchQuery.setSort(new Sort(Arrays.asList(fieldSort)));
Multi-valued fields
For multi-valued fields, such as array type or nested type fields, you can use the mode parameter to specify which element to use for sorting.
Sort by a specified value in a multi-valued array.
// Assume you have two rows, doc1 and doc2. Both have a field1 of array type.
// The value of field1 in doc1 is [2,3]. The value of field1 in doc2 is [1,3,4].
// You can set the mode parameter to specify which value in the array to use for sorting.
{
// When mode is set to SortMode.MAX, the sort order is doc2 (sorted by 4), then doc1 (sorted by 3).
FieldSort fieldSort = new FieldSort("field1", SortOrder.DESC);
fieldSort.setMode(SortMode.MAX);
}
{
// When mode is set to SortMode.MIN, the sort order is doc1 (sorted by 2), then doc2 (sorted by 1).
FieldSort fieldSort = new FieldSort("field1", SortOrder.DESC);
fieldSort.setMode(SortMode.MIN);
}You can also sort the sub-rows of a nested type field.
// Assume you have two rows, doc1 and doc2. Both have a field1 of nested type.
// The value of field1 in doc1 is [{"name":"b", "age":1},{"name":"a", "age":7}].
// The value of field1 in doc2 is [{"name":"a", "age":1},{"name":"c", "age":1},{"name":"d", "age":5}].
{
// Sort all sub-rows and use the mode parameter to specify which value to use for sorting.
// When mode is set to SortMode.MAX and you sort by the age field, the result is doc1 (sorted by 7), then doc2 (sorted by 5).
FieldSort fieldSort = new FieldSort("field1.age", SortOrder.DESC);
fieldSort.setMode(SortMode.MAX);
String path = "field1";
NestedFilter nestedFilter = new NestedFilter(path, QueryBuilders.matchAll().build());
fieldSort.setNestedFilter(nestedFilter);
}
{
// Sort only the sub-rows where age=1, and use the mode parameter to specify which value to use.
{
// When mode is set to SortMode.MAX and you sort by the name field, the result is doc2 (sorted by "c"), then doc1 (sorted by "b").
FieldSort fieldSort = new FieldSort("field1.name", SortOrder.DESC);
fieldSort.setMode(SortMode.MAX);
String path = "field1";
NestedFilter nestedFilter = new NestedFilter(path, QueryBuilders.term("field1.age",1).build());
fieldSort.setNestedFilter(nestedFilter);
}
{
// When mode is set to SortMode.MIN and you sort by the name field, the result is doc1 (sorted by "b"), then doc2 (sorted by "a").
FieldSort fieldSort = new FieldSort("field1.name", SortOrder.DESC);
fieldSort.setMode(SortMode.MIN);
String path = "field1";
NestedFilter nestedFilter = new NestedFilter(path, QueryBuilders.term("field1.age",1).build());
fieldSort.setNestedFilter(nestedFilter);
}
}GeoDistanceSort
Sorts results by the distance from a geographic point.
SearchQuery searchQuery = new SearchQuery();
// The 'geo' field is of the GeoPoint type. Sort results by the distance
// from the value in this field to the point "0,0".
Sort.Sorter sorter = new GeoDistanceSort("geo", Arrays.asList("0, 0"));
searchQuery.setSort(new Sort(Arrays.asList(sorter)));Paging methods
To page through query results, you can use limit and offset parameters or a token.
Offset-based paging
You can use offset-based paging when the total number of rows to retrieve is less than 100,000. The sum of limit and offset must be less than or equal to 100,000, and the maximum value for limit is 100.
To increase the limit threshold, see How do I increase the limit for the Search API to 1,000?.
If you do not set the limit and offset parameters, limit defaults to 10 and offset defaults to 0.
SearchQuery searchQuery = new SearchQuery();
searchQuery.setQuery(new MatchAllQuery());
searchQuery.setLimit(100);
searchQuery.setOffset(100);Token-based paging
Token-based paging is recommended for deep paging because it has no depth limit.
If a response does not contain all matching data, the server returns a nextToken.
By default, token-based paging only allows you to move forward through the results. However, because a token remains valid throughout the query process, you can cache a previous token to page backward.
If you need to persist a nextToken or send it to a front-end application, use Base64 to encode it into a string. A token is a byte array, not a string. Directly converting it to a string with new String(nextToken) causes data loss.
When you page with a token, the sort order is the same as in the previous request, whether it uses the default IndexSort or a custom sort. Therefore, you cannot set a Sort parameter when a token is used. You also cannot set an offset, as you can only read data sequentially.
A search index that contains a nested type field does not support index presorting. If you need to page through results from such an index, you must specify a sort order in the query. Otherwise, the server does not return a nextToken even if more data is available.
private static void readMoreRowsWithToken(SyncClient client) {
SearchQuery searchQuery = new SearchQuery();
searchQuery.setQuery(new MatchAllQuery());
searchQuery.setGetTotalCount(true);// Set to return the total count of matched rows.
// Specify the data table name (for example, sampleTable) and the search index name (for example, sampleSearchIndex).
// You can find the search index name on the Index Management tab for the table in the Tablestore console, or by listing search indexes with the SDK.
SearchRequest searchRequest = new SearchRequest("sampleTable", "sampleSearchIndex", searchQuery);
SearchResponse resp = client.search(searchRequest);
if (!resp.isAllSuccess()) {
throw new RuntimeException("not all success");
}
List<Row> rows = resp.getRows();
while (resp.getNextToken()!=null) { // A null nextToken means all data has been read.
// Get the nextToken.
byte[] nextToken = resp.getNextToken();
{
// If you need to persist the nextToken or send it to a front-end application,
// use Base64 to encode the nextToken into a string for storage and transfer.
// A token is a byte array. Directly using new String(nextToken) will cause data loss.
String tokenAsString = Base64.toBase64String(nextToken);
// Decode the string back to a byte array.
byte[] tokenAsByte = Base64.fromBase64String(tokenAsString);
}
// Set the token for the next request.
searchRequest.getSearchQuery().setToken(nextToken);
resp = client.search(searchRequest);
if (!resp.isAllSuccess()) {
throw new RuntimeException("not all success");
}
rows.addAll(resp.getRows());
}
System.out.println("RowSize: " + rows.size());
System.out.println("TotalCount: " + resp.getTotalCount());// Prints the total number of matched rows, not the number of returned rows.
}