layer clause

Clause description

Hierarchical queries extend the query syntax to provide more control over document retrieval. This lets you speed up document retrieval and improve system performance for your application. The main features are:

• You can select a range for retrieval.

• Set the retrieval order for different ranges.

• Use different queries for different ranges to affect the retrieval set.

Syntax

Terms

Query syntax

{
  "layer" : [
  ]
}

Select retrieval ranges

You can use the layer syntax to control retrieval ranges.

A layer clause can have multiple single_layer parts. A single_layer has two main elements:

• quota: Sets the recommended number of documents to retrieve for the current layer. Note the following:

  • Relationship between quota and rank_size: The sum of quotas for all layers cannot be greater than the rank_size value. For example, if rank_size is 10, and you set quota:5 for the first layer and quota:7 for the second, the quota for the second layer is reduced to 5.

  • If a layer has a remaining quota, it is automatically added to the next layer's quota.

  • Documents in different ranges within a layer are not always ordered by relevance. Because of this, quota works in two ways. One way is to check the remaining quota after each document is found. The other way is to ignore the quota limit within the current layer, but still respect the rank_size limit. After the engine searches the entire layer, it adds any remaining quota to the next layer.

  • If you explicitly specify a layer clause, the default quota for each layer is 0. The maximum value for quota is the maximum value of uint32_t.

• range: Determines the document scope for the current layer. If you do not specify a range, the scope includes all documents, which is [0, docCount). The range syntax uses the attributes you provide to calculate the final document range to seek. Note the following:

  • The syntax must use attributes, not calculated expressions.

  • The attributes used in the syntax must match the offline sorting method. If they do not match, the query searches the entire range.

  • The attributes must be consecutive. Do not insert extension keywords, such as %sorted or %docid, between attributes.

  • In addition to attributes, several extension statements are supported. The %sorted keyword queries sorted full and incremental data in this layer. The %unsorted keyword queries unsorted data in this layer, such as real-time data. The %other keyword queries the range outside of the preceding layers. The %docid keyword directly specifies a docid range to query. The %segmentid keyword directly specifies a segment ID range to query. The %percent keyword specifies a percentage-based range to query.

  • If you do not specify the %sorted or %unsorted keywords in the layer clause, the engine adds them automatically. By default, the engine sorts each layer. If not enough results are returned, the engine queries an additional layer of real-time data.

Select different queries for different ranges

For different query layers, you can use different search queries or select different posting lists. Separate different queries with a semicolon (;). If the number of layers is greater than the number of queries, the last query is automatically used for the remaining layers.

Examples

This section provides examples of how to use hierarchical queries with other engine features in different scenarios.

Sort documents during index building and query a specified retrieval range

When you use the offline sorting feature, documents are globally ordered. For example, after sorting by site, documents from the same site form a continuous block in the index. When you query within a specific site, you can use this information to speed up the seek operation. Assume that the attribute for the site is site_id. To query for "iphone" on sites with IDs 1 and 7:

{
  "layer" : [
    {
      "range" : { 
        "fields" : [
          {
            "field" : "site_id",
            "values" : [1,7]
          }
        ]
      },
      "quota" : 5000
    }
  ]
}

If sites 1 and 7 do not return enough results, you can query sites 5 and 10 for more results:

{
  "layer" : [
    {
      "range" : { 
        "fields" : [
          {
            "field" : "site_id",
            "values" : [1,7]
          }
        ]
      },
      "quota" : 5000
    },
    {
      "range" : { 
        "fields" : [
          {
            "field" : "site_id",
            "values" : [5,10]
          }
        ]
      },
      "quota" : 0
    }
  ]
}

The second layer is queried only if the first layer does not return enough results. You can set the quota for the second layer to 0 or leave it unspecified. You can also perform a diverse retrieval:

{
  "layer" : [
    {
      "range" : { 
        "fields" : [
          {
            "field" : "site_id",
            "values" : [1,7]
          }
        ]
      },
      "quota" : 4000
    },
    {
      "range" : { 
        "fields" : [
          {
            "field" : "site_id",
            "values" : [5,10]
          }
        ]
      },
      "quota" : 1000
    }
  ]
}

You can define multi-dimensional ranges for multi-dimensional offline sorting. For example, consider an intra-site query where documents are first sorted by site. Documents on the same site are then sorted by the webpage's static score. To retrieve webpages with a static score greater than 100, use the following query syntax:

{
  "layer" : [
    {
      "range" : { 
        "fields" : [
          {
            "field" : "site_id",
            "values" : [1,7]
          },
          {
            "field" : "static_score",
            "values" : "[100,]"
          }
        ]
      },
      "quota" : 4000
    }
  ]
}

Note: The range is enclosed in double quotes.

Multi-query search

In some cases, you may want to retrieve a sufficient number of documents, but not so many that it degrades query performance. A common scenario involves multiple search queries, such as A and B:

The query patterns above yield varying numbers of retrieved documents and different performance levels. For most queries, the goal is to achieve effective retrieval without an excessive number of documents. However, with a fixed query pattern, it is difficult to balance the number of results with performance. You can use a multi-query approach to address this trade-off within a single query:

{
  "query": "A OR B;A RANK B;A AND B",
  "layer" : [
    {
      "quota" : 1000
    },
    {
      "quota" : 1000
    },
    {
      "quota" : 1000
    }
  ]
}

This query method balances the number of retrieved documents with performance and significantly improves search relevance. For large retrievals, the method can hit results that match both A and B. For small retrievals, it uses an A OR B approach to retrieve as many documents as possible.

Time-sensitive queries

If you want to prioritize the timeliness of query results, you can first query the real-time, unordered index, and then the full and incremental ordered indexes. Within an ordered index, you can also use the percent parameter to query more recent data before older data.

For example, for the search query 'iPhone', you can first query the most recent real-time index. If the results are insufficient, you can then query the last 50% of documents in the ordered index, and finally query the first 50% of documents.

{
  "layer" : [
    {
      "range" : { 
        "index_type" : "%unsorted"
      },
      "quota" : 5000
    },
    {
      "range" : { 
        "index_type" : "%sorted",
        "fields" : [
          {
            "field" : "service_id",
            "values" : [1,3]
          }
        ],
        "percent" : "[50,100)"
      },
      "quota" : 0
    },
    {
      "range" : { 
        "index_type" : "%sorted",
        "fields" : [
          {
            "field" : "service_id",
            "values" : [1,3]
          }
        ],
        "percent" : "[0,50)"
      },
      "quota" : 0
    }
  ]
}

Note: In the percent keyword, you can specify multiple intervals, which are left-inclusive and right-exclusive.