Sorting and paging

Updated at:

When querying data with a search index, you can apply a predefined sort order or specify one at query time. To navigate large result sets, use limit/offset for offset-based paging, or use tokens for cursor-based paging.

Scenarios

Category

Method

Feature

Scenario

Sorting

Predefine a sorting method when you create a search index

Index presorting (IndexSort)

By default, results are returned in the order defined by the IndexSort parameter set during index creation.

Specify a sorting method when you query data

Sorting based on the BM25-based keyword relevance score (ScoreSort)

Sort results by BM25 relevance score. Best suited for full-text search.

Sorting based on the primary key value (PrimaryKeySort)

Sort results by primary key value. Useful when you need results ordered by unique row identifiers.

Sorting based on the values of one or more columns (FieldSort)

Sort results by one or more column values, such as sales volume or page views. Commonly used in e-commerce, social networking, and media asset management.

Sorting by geographical location (GeoDistanceSort)

Sort results by distance from a central point. Commonly used in mapping and logistics—for example, sorting nearby restaurants by distance.

Paging

Specify a paging method when you query data

Paging based on the limit and offset parameters

Jump to any page when the result set contains fewer than 100,000 rows.

Paging based on tokens

Page through results sequentially using a cursor; you can only page forward. Tokens remain valid for the duration of the query, so you can cache a previous token to return to an earlier page.

Index presorting

By default, Tablestore sorts data in a search index based on a predefined sort order, called index presorting (IndexSort). When you query data, IndexSort determines the default order of the returned results.

When you create a search index, you can customize IndexSort. If you do not specify this setting, the index defaults to primary key sort.

Important
  • Index presorting supports only PrimaryKeySort (primary key sort) and FieldSort (field sort).

  • You cannot use index presorting for a search index that contains nested type fields.

  • After you create a search index, the dynamic schema modification feature enables you to modify the IndexSort settings.

Specify a sorting method

Sorting requires enableSortAndAgg to be set to true on the fields you want to sort by.

You can specify a sorting method for each query. A search index supports the following four types of sorters. You can also use multiple sorters to sort results based on a sequence of criteria.

ScoreSort

Sorts query results by relevance score, which is calculated using the BM25 algorithm. This method is suitable for relevance-based scenarios, such as full-text search.

Important
  • If you want to sort by relevance score, you must explicitly specify ScoreSort. Otherwise, Tablestore sorts the results according to the IndexSort settings of the index.

  • When you use ScoreSort, fields of the FuzzyKeyword type are not included in the sorting, and the weight parameter has no effect on these fields.

Use ScoreSort to rank results by BM25 relevance score, in ascending or descending order.

sort: {
    sorters: [
        {
            scoreSort: {
                order: TableStore.SortOrder.SORT_ORDER_ASC
            }
        }
    ]
}

PrimaryKeySort

Sorts results by primary key.

Use PrimaryKeySort to order results by primary key value.

sort: {
    sorters: [
        {
            primaryKeySort: {
                order: TableStore.SortOrder.SORT_ORDER_DESC // Sort in descending order.
                //order: TableStore.SortOrder.SORT_ORDER_ASC // Sort in ascending order.
            }
        }
    ]
}

FieldSort

Use FieldSort to sort results by one or more column values.

Sort by a single column

sort: {
    sorters: [
        {
            fieldSort: {
                fieldName: "Col_Keyword",
                order: TableStore.SortOrder.SORT_ORDER_DESC
            }
        }
    ]
}

Sort by multiple columns

Specify multiple sorters to sort by a primary column first, then break ties using a secondary column.

sort: {
    sorters: [
        {
            fieldSort: {
                fieldName: "Col_Keyword",
                order: TableStore.SortOrder.SORT_ORDER_DESC
            }
        },
        {
            fieldSort: {
                fieldName: "Col_Long",
                order: TableStore.SortOrder.SORT_ORDER_DESC
            }
        }
    ]
}

GeoDistanceSort

Sorts results by the distance from a geographic point.

Use GeoDistanceSort to rank results by distance from a central geographic point.

sort: {
    sorters: [
        {
            geoDistanceSort: {
                fieldName: "Col_Geo_Point",
                points: ["0,0"],// Specify the coordinate pair of the central point.
                order: TableStore.SortOrder.SORT_ORDER_ASC // Return results in ascending order of distance.
            }
        }
    ]
}

For a complete example, see Search on GitHub.

Specify a paging method

To page through query results, you can use limit and offset parameters or a token.

Configure the limit and offset parameters

Offset-based paging

You can use offset-based paging when the total number of rows to retrieve is less than 100,000. The sum of limit and offset must be less than or equal to 100,000, and the maximum value for limit is 100.

Note

To increase the limit threshold, see How do I increase the limit for the Search API to 1,000?.

If you do not set the limit and offset parameters, limit defaults to 10 and offset defaults to 0.

Use offset and limit to jump directly to any page. This method supports result sets of up to 100,000 rows.

/**
 * Set offset to 90 and limit to 10 to retrieve rows 90–99.
 */
client.search({
    tableName: TABLE_NAME,
    indexName: INDEX_NAME,
    searchQuery: {
        offset: 90,
        limit: 10,
        query: {
            queryType: TableStore.QueryType.MATCH_ALL_QUERY
        },
        getTotalCount: true // Return the total number of matching rows. Default: false.
    },
    columnToGet: {
        // RETURN_ALL: return all columns.
        // RETURN_SPECIFIED: return specified columns.
        // RETURN_NONE: return primary key columns only.
        returnType: TableStore.ColumnReturnType.RETURN_ALL
    }
}, function (err, data) {
    if (err) {
        console.log('error:', err);
        return;
    }
    console.log('success:', JSON.stringify(data, null, 2));
});

Use a token

Token-based paging is recommended for deep paging because it has no depth limit.

If a response does not contain all matching data, the server returns a nextToken.

By default, token-based paging only allows you to move forward through the results. However, because a token remains valid throughout the query process, you can cache a previous token to page backward.

Token-based paging returns results page by page using a cursor (NextToken). Each response includes a token for the next page. Tokens are valid for the duration of the query, so cache a previous token to navigate back to an earlier page.

Important

To persist or pass NextToken to a frontend page, encode it as a Base64 string. Tokens are binary data (byte stream), not strings—using string(NextToken) loses data. Convert using Buffer:

  1. Encode: data.nextToken.toString("base64")

  2. Decode: Buffer.from(base64String, "base64")

When you page with a token, the sort order is the same as in the previous request, whether it uses the default IndexSort or a custom sort. Therefore, you cannot set a Sort parameter when a token is used. You also cannot set an offset, as you can only read data sequentially.

Important

A search index that contains a nested type field does not support index presorting. If you need to page through results from such an index, you must specify a sort order in the query. Otherwise, the server does not return a nextToken even if more data is available.

The following examples show token-based paging in both synchronous and asynchronous modes. Both use the same initial params object.

var params = {
    tableName: TABLE_NAME,
    indexName: INDEX_NAME,
    searchQuery: {
        offset: 0,
        limit: 10,
        token: null, // Set to nextToken from the previous response to fetch the next page.
        query: {
            queryType: TableStore.QueryType.MATCH_ALL_QUERY
        },
        getTotalCount: true
    },
    columnToGet: {
        returnType: TableStore.ColumnReturnType.RETURN_SPECIFIED,
        returnNames: ["pic_tag", "pic_description", "time_stemp", "pos"]
    }
};

/**
 * Synchronous mode: await each page before fetching the next.
 */
(async () => {
  try {
    var data = await client.search(params);
    console.log('success:', JSON.stringify(data, null, 2));

    while (data.nextToken && data.nextToken.length) {
      // Encode the binary token as Base64 for storage or transfer,
      // then decode it back to binary before passing it as the next token.
      var nextToken = data.nextToken.toString("base64");
      var token = Buffer.from(nextToken, "base64");

      params.searchQuery.token = token;
      data = await client.search(params);
      console.log('token success:', JSON.stringify(data, null, 2));
    }
  } catch (error) {
      console.log(error);
  }
})()

/**
 * Asynchronous mode: use callbacks to fetch the next page.
 */
client.search(params, function (err, data) {
    console.log('success:', JSON.stringify(data, null, 2));

    if (data.nextToken && data.nextToken.length) {
        // Encode and decode the token the same way as in synchronous mode.
        var nextToken = data.nextToken.toString("base64");
        var token = Buffer.from(nextToken, "base64");

        params.searchQuery.token = token;
        client.search(params, function (err, data) {
            console.log('token success:', JSON.stringify(data, null, 2));
        });
    }
});