Summaries and highlighting

Updated at:

To use the summary and highlighting feature, you must enable highlighting for a Text field when you create a search index. Then, you can set highlight parameters in your query to return snippets that contain the matched search queries and highlight the matched terms.

Scenarios

You can use the summary and highlighting feature to extract snippets near matched search queries and highlight the queries. This feature is useful in full-text search scenarios, such as web searches, chat record retrieval, and document searches.

Function overview

The summary and highlighting feature highlights text in search results that matches or is relevant to your search queries. This helps users quickly find relevant content and improves information retrieval efficiency. This feature is also supported for complex data structures that contain nested types, such as JSON, to precisely locate information. By default, Tablestore uses the <em></em> tags to mark matched search queries.

To use the summary and highlighting feature, you must perform the following configurations:

  1. When you create a search index, set the enableHighlighting parameter to True for the Text field. For more information, see Create a search index.

    Important

    You can enable the summary and highlighting feature only for Text fields.

  2. When you query data, you can customize the highlight style by specifying parameters, such as the encoding method for highlighted fragments, the maximum number of fragments to return, and the pre-tags and post-tags.

Assume that you have enabled the summary and highlighting feature for a Text field when you created a search index. If the value of the Text field is query highlight test, you can use a match phrase query for the keyword highlight. After you specify the highlight parameters, the highlighted fragment query <em>highlight</em> test is returned.

Notes

  • When you use the summary and highlighting feature with a MatchQuery or MatchPhraseQuery, the matched terms may be highlighted by multiple pre-tags and post-tags.

  • If the tokenizer type for the Text field is maximum semantic tokenization, the summary and highlighting feature is not supported when you use a MatchPhraseQuery to query data.

  • Fragment splitting may divide a matched term in the text. In this case, the term may not be highlighted.

API

The summary and highlighting feature uses the Search API. The supported query types are TermQuery, TermsQuery, MatchQuery, MatchPhraseQuery, PrefixQuery, WildcardQuery, and NestedQuery.

Parameters

When you use the summary and highlighting feature, you can set highlight parameters using the Highlight parameter. For sub-columns of a nested type, you can set the highlight parameters using the InnerHits parameter.

Highlight parameters

Name

Type

Description

fieldHighlightParams (required)

Map<String, HighlightParameter>

The field names and fragment settings. A key is the name of a field whose summary and highlighting results you want to return. Its value is the HighlightParameter object for the field. The field must have summary and highlighting enabled and must be used in a supported query condition.

highlightEncoder (optional)

HighlightEncoder

The encoding method for original text in highlighted fragments. Valid values:

  • PLAIN (default): Does not encode original text.

  • HTML: HTML-escapes original text. Use this value if you display results on a web page. The characters <, >, ", ', and / are escaped as &lt;, &gt;, &quot;, &#x27;, and &#x2F;, respectively.

fieldHighlightParams[].numberOfFragments (optional)

Integer

The maximum number of highlighted fragments to return for a field. We recommend that you set this parameter to 1.

fieldHighlightParams[].fragmentSize (optional)

Integer

The target length of each fragment. The default value is 100. The actual fragment length may differ from this value.

fieldHighlightParams[].preTag (optional)

String

The pre-tag for a matched term. The default value is <em>. You can specify a custom tag such as <b>. You must configure preTag and postTag together. The supported characters are < > " ' /, a-z, A-Z, and 0-9.

fieldHighlightParams[].postTag (optional)

String

The post-tag for a matched term. The default value is </em>. You can specify a custom tag such as </b>. You must configure preTag and postTag together. The supported characters are the same as those for preTag.

fieldHighlightParams[].highlightFragmentOrder (optional)

HighlightFragmentOrder

The order in which multiple highlighted fragments are returned. TEXT_SEQUENCE (default) sorts fragments by their positions in the original text. SCORE sorts fragments by the relevance scores of matched terms.

InnerHits parameters

Parameter

Description

sort

The sorting rule for the returned nested sub-rows.

offset

The starting position from which to return sub-rows when a nested column contains multiple sub-rows.

limit

The number of sub-rows to return when a nested column contains multiple sub-rows. Default value: 3.

highlight

The highlight parameter configuration for nested sub-columns. For more information about the configuration, see Highlight parameters.

Usage

Important

You can use the summary and highlighting feature only through an SDK.

Before you can perform a highlight query, you must complete the following preparations:

  • You must have an Alibaba Cloud account or a RAM user that has the permissions to perform operations on Tablestore. To grant a RAM user the permissions to perform operations on Tablestore, see Grant permissions to a RAM user using a RAM policy.

    When you use an SDK, if you do not have a valid AccessKey, you must create one for your Alibaba Cloud account or RAM user. For more information, see Create an AccessKey.

  • A data table has been created. For more information, see Data table operations.

  • Create a search index for the data table and enable highlighting for the specified field. For more information, see Create a search index.

  • When you use an SDK, you must initialize the Tablestore client. For more information, see Initialize Tablestore Client.

You can use the summary and highlighting feature with the Java SDK, Go SDK, Python SDK, and Node.js SDK. The following examples show how to use this feature with the Java SDK.

Using query summary and highlighting for non-nested fields

The following example uses a match query to retrieve rows in which the description field contains the tablestore term. The example encloses matched terms in the returned fragments with <b> and </b>. The description field is a Text field for which summary and highlighting is enabled.

String tableName = "example_table";
String indexName = "example_index";

MatchQuery matchQuery = new MatchQuery();
matchQuery.setFieldName("description");
matchQuery.setText("tablestore");

HighlightParameter highlightParameter = new HighlightParameter();
highlightParameter.setPreTag("<b>");
highlightParameter.setPostTag("</b>");

Highlight highlight = new Highlight();
highlight.addFieldHighlightParam("description", highlightParameter);

SearchQuery searchQuery = new SearchQuery();
searchQuery.setQuery(matchQuery);
searchQuery.setHighlight(highlight);
searchQuery.setLimit(10);

SearchRequest request = new SearchRequest(tableName, indexName, searchQuery);
SearchResponse response = client.search(request);
for (SearchHit hit : response.getSearchHits()) {
    HighlightResultItem resultItem = hit.getHighlightResultItem();
    if (resultItem == null) {
        continue;
    }
    HighlightField field = resultItem.getHighlightFieldByName("description");
    if (field != null) {
        System.out.println(field.getFragments());
    }
}

Use highlighting when you query nested fields

The following example shows how to use `NestedQuery` to query for data where the value of the `Level1_Col1_Nested` sub-field in the `Col_Nested` nested field matches `hangzhou shanghai. The search query is highlighted in the returned results.

MatchQuery matchQuery = new MatchQuery();
matchQuery.setFieldName("items.description");
matchQuery.setText("hangzhou");

HighlightParameter parameter = new HighlightParameter();
parameter.setPreTag("<em>");
parameter.setPostTag("</em>");
Highlight highlight = new Highlight();
highlight.addFieldHighlightParam("items.description", parameter);

InnerHits innerHits = new InnerHits();
innerHits.setLimit(3);
innerHits.setSort(new Sort(Arrays.asList(
        new ScoreSort(), new DocSort(SortOrder.ASC))));
innerHits.setHighlight(highlight);

NestedQuery nestedQuery = new NestedQuery();
nestedQuery.setPath("items");
nestedQuery.setQuery(matchQuery);
nestedQuery.setScoreMode(ScoreMode.None);
nestedQuery.setInnerHits(innerHits);

SearchQuery searchQuery = new SearchQuery();
searchQuery.setQuery(nestedQuery);

Assume that the multi-level nested field `Col_Nested` includes two sub-fields: `Level1_Col1_Text` (Text) and `Level1_Col2_Nested` (Nested). The `Level1_Col2_Nested` nested field includes the `Level2_Col1_Text` sub-field.

The following example shows how to add a `BoolQuery` to a `NestedQuery` to use the summary and highlighting features on both the `Level1_Col1_Text` sub-field in the `Col_Nested` field and the `Level2_Col1_Text` sub-field under `Level1_Col2_Nested`.

public static void nestedQueryWithHighlighting(SyncClient client) {
    SearchRequest searchRequest = SearchRequest.newBuilder()
            .tableName("<TABLE_NAME>")
            .indexName("<SEARCH_INDEX_NAME>")
            .returnAllColumnsFromIndex(true)
            .searchQuery(SearchQuery.newBuilder()
                    .limit(5)
                    .query(QueryBuilders.nested()
                            .path("Col_Nested")
                            .scoreMode(ScoreMode.Min)
                            .query(QueryBuilders.bool()
                                    .should(QueryBuilders.match("Col_Nested.Level1_Col1_Text", "hangzhou shanghai"))
                                    .should(QueryBuilders.nested()
                                            .path("Col_Nested.Level1_Col2_Nested")
                                            .scoreMode(ScoreMode.Min)
                                            .query(QueryBuilders.match("Col_Nested.Level1_Col2_Nested.Level2_Col1_Text", "hangzhou shanghai"))
                                            .innerHits(InnerHits.newBuilder()
                                                    .highlight(Highlight.newBuilder()
                                                            .addFieldHighlightParam("Col_Nested.Level1_Col2_Nested.Level2_Col1_Text", HighlightParameter.newBuilder().build())
                                                            .build())
                                                    .build())))
                            .innerHits(InnerHits.newBuilder()
                                    .sort(new Sort(Arrays.asList(
                                            new ScoreSort(),
                                            new DocSort()
                                    )))
                                    .highlight(Highlight.newBuilder()
                                            .addFieldHighlightParam("Col_Nested.Level1_Col1_Text", HighlightParameter.newBuilder().build())
                                            .build())
                                    .build()))
                    .build())
            .build();
    SearchResponse resp = client.search(searchRequest);
    // Print the highlighted results.
    printSearchHit(resp.getSearchHits(), "");
}

/**
 * Print the content of searchHit.
 * @param searchHits The search hits.
 * @param prefix The prefix to add when printing nested structures to show hierarchical information.
 */
private static void printSearchHit(List<SearchHit> searchHits, String prefix) {
    for (SearchHit searchHit : searchHits) {
        if (searchHit.getScore() != null) {
            System.out.printf("%s Score: %s
", prefix, searchHit.getScore());
        }

        if (searchHit.getOffset() != null) {
            System.out.printf("%s Offset: %s
", prefix, searchHit.getOffset());
        }

        if (searchHit.getRow() != null) {
            System.out.printf("%s Row: %s
", prefix, searchHit.getRow().toString());
        }

        // Print the highlighted fragments for each field.
        if (searchHit.getHighlightResultItem() != null) {
            System.out.printf("%s Highlight: 
", prefix);
            StringBuilder strBuilder = new StringBuilder();
            for (Map.Entry<String, HighlightField> entry : searchHit.getHighlightResultItem().getHighlightFields().entrySet()) {
                strBuilder.append(entry.getKey()).append(":").append("[");
                strBuilder.append(StringUtils.join(",", entry.getValue().getFragments())).append("]
");
            }
            System.out.printf("%s   %s", prefix, strBuilder);
        }

        // Highlighted results for the nested type.
        for (SearchInnerHit searchInnerHit : searchHit.getSearchInnerHits().values()) {
            System.out.printf("%s Path: %s
", prefix, searchInnerHit.getPath());
            System.out.printf("%s InnerHit: 
", prefix);
            printSearchHit(searchInnerHit.getSubSearchHits(), prefix + "    ");
        }

        System.out.println();
    }
}

Billing

Using the summary and highlighting feature during data queries does not affect the existing billing rules.

In VCU mode (formerly reserved mode), querying data using a search index consumes the compute resources of the VCU. In CU mode (formerly pay-as-you-go mode), querying data using a search index consumes read throughput. For more information, see Search index metering and billing.

FAQ

References