layer clause

Clause description

Layered queries extend the query syntax and give you more control over the document retrieval process. Applications can use layered queries to accelerate document retrieval and improve overall system performance. The main extended features are:

• Select ranges for document retrieval.

• Set the retrieval order for different ranges.

• Specify different queries for different ranges to influence the retrieval set.

Clause syntax

Terminology

Query syntax

{
  "layer" : [
  ]
}

Select retrieval ranges

You can use the layer syntax to control retrieval ranges.

A layer clause consists of one or more `single_layer` elements. Each `single_layer` element has two main components:

• quota: Sets the recommended number of documents to retrieve for the current layer. Note the following points:

  • Relationship between quota and `rank_size`: The sum of quotas for all layers cannot exceed `rank_size`. For example, if `rank_size` is 10 and you have `quota:5` and `quota:7`, the quota for the second layer is truncated to 5.

  • If the current layer has a remaining quota, the quota is automatically added to the next layer.

  • The quota can be applied in two ways because documents within a layer are not always sorted by relevance. The first way checks the remaining quota after each document is found. The second way ignores the quota limit within the current layer but still respects the `rank_size` limit. After the current layer is searched, any remaining quota is added to the next layer.

  • If you explicitly specify a layer clause, the default quota for each layer is 0. The maximum value for quota is the maximum value of `uint32_t`.

• range: Determines the document range for the current layer. If this parameter is not specified, the range includes all documents, which is `[0, docCount)`. The `range` syntax progressively calculates the required document range to seek based on the attributes that you provide. Note the following:

  • You must use attributes in the syntax, not expressions that require calculation.

  • The attributes used in the syntax must match the offline sorting method. Otherwise, the query automatically searches the entire range.

  • The attributes used in the syntax must appear consecutively. You cannot insert extension keywords such as `%sorted` or `%docid` between the attributes.

  • In addition to attributes, several extension statements are supported. These include `%sorted` to query the sorted full and incremental indexes in this layer, `%unsorted` to query unsorted data such as real-time data in this layer, `%other` for the range that excludes previous layers, `%docid` to directly specify a docid range to query, `%segmentid` to directly specify a segment ID range to query, and `%percent` to specify a percentage of the range to query.

  • If you do not specify the `%sorted` or `%unsorted` keywords in the layer clause, the engine automatically adds them. By default, each layer is sorted. If an insufficient number of results are returned, the engine queries an additional layer of real-time data.

Select different queries for different ranges

For different query layers, you can use different search queries or select different inverted lists by separating the queries with semicolons. If the number of layers is greater than the number of queries, the last query is automatically used for the remaining layers.

Usage examples

This section describes how to use layered queries with other features that are provided by the engine.

Sort documents during index building and query specified retrieval ranges

When you use the offline sorting feature, documents are globally ordered. For example, if you sort by site, all documents from the same site are grouped together in the index. You can use this feature to accelerate seek operations when you query within a site. For example, assume that the site attribute is `site_id`. To search for "iphone" within site IDs 1 and 7, use the following syntax:

{
  "layer" : [
    {
      "range" : { 
        "fields" : [
          {
            "field" : "site_id",
            "values" : [1,7]
          }
        ]
      },
      "quota" : 5000
    }
  ]
}

Alternatively, if the query results from sites 1 and 7 are insufficient and you want to query sites 5 and 10 for additional results, use the following syntax:

{
  "layer" : [
    {
      "range" : { 
        "fields" : [
          {
            "field" : "site_id",
            "values" : [1,7]
          }
        ]
      },
      "quota" : 5000
    },
    {
      "range" : { 
        "fields" : [
          {
            "field" : "site_id",
            "values" : [5,10]
          }
        ]
      },
      "quota" : 0
    }
  ]
}

If you want to query the second layer only when the results from the first layer are insufficient, you can set the quota for the second layer to 0 or leave it unspecified. You can also perform diversity retrieval:

{
  "layer" : [
    {
      "range" : { 
        "fields" : [
          {
            "field" : "site_id",
            "values" : [1,7]
          }
        ]
      },
      "quota" : 4000
    },
    {
      "range" : { 
        "fields" : [
          {
            "field" : "site_id",
            "values" : [5,10]
          }
        ]
      },
      "quota" : 1000
    }
  ]
}

For multi-dimensional offline sorting, you can also locate multi-dimensional ranges. For example, in an intra-site search, offline sorting is first performed by site. For webpages on the same site, the webpages are then sorted by their static score. In this case, to retrieve webpages that have a static score greater than 100, the query syntax is as follows:

{
  "layer" : [
    {
      "range" : { 
        "fields" : [
          {
            "field" : "site_id",
            "values" : [1,7]
          },
          {
            "field" : "static_score",
            "values" : "[100,]"
          }
        ]
      },
      "quota" : 4000
    }
  ]
}

Note: Enclose interval ranges in double quotation marks.

Multi-query

In some cases, users want to retrieve a sufficient number of items for a query but also want to avoid retrieving too many items, which can negatively affect query performance. A common scenario for this is when a query contains multiple search terms, such as A and B:

The preceding query methods yield different retrieval counts and performance for A and B. Most queries require good retrieval, but the retrieval count does not need to be high. In this scenario, it is difficult to use a single, fixed query method to ensure that both the number of results and performance are moderate. A multi-query approach can solve this issue in a single query:

{
  "query": "A OR B;A RANK B;A AND B",
  "layer" : [
    {
      "quota" : 1000
    },
    {
      "quota" : 1000
    },
    {
      "quota" : 1000
    }
  ]
}

This query method balances document recall and performance. It also significantly improves the relevance of the search query. For large-scale retrieval, it can hit some 'A AND B' results. For small-scale retrieval, it can use 'A OR B' to recall as many documents as possible.

Time-sensitive queries

If you want to prioritize the recency of query results, you can first query the real-time unordered index before you query the full and incremental ordered indexes. Within an ordered index, you can also use the percent parameter to specify the query order, such as querying a later portion of data before an earlier portion.

For example, consider a scenario where you want to search for 'iphone'. You first query the most up-to-date real-time index. If the results are insufficient, you then query the latter 50% of documents in the ordered index. Finally, you query the first 50% of documents:

{
  "layer" : [
    {
      "range" : { 
        "index_type" : "%unsorted"
      },
      "quota" : 5000
    },
    {
      "range" : { 
        "index_type" : "%sorted",
        "fields" : [
          {
            "field" : "service_id",
            "values" : [1,3]
          }
        ],
        "percent" : "[50,100)"
      },
      "quota" : 0
    },
    {
      "range" : { 
        "index_type" : "%sorted",
        "fields" : [
          {
            "field" : "service_id",
            "values" : [1,3]
          }
        ],
        "percent" : "[0,50)"
      },
      "quota" : 0
    }
  ]
}

Note: You can specify multiple intervals for the `percent` keyword. The intervals are left-closed and right-open.

Notes

• The layer clause is optional.