Phrase query (position index)-Simple Log Service(SLS)-阿里云帮助中心

A position index records the order and position of tokens, enabling phrase match at the index level. After you enable a position index, phrase queries return precise log results and histograms without scanning candidate logs.

Important

The position index feature is currently in preview and is free of charge. This feature is available only to users on the allowlist. To use this feature, submit a ticket to apply for access.

How it works

A position index is a Logstore-level switch that inherits the existing index configurations for case sensitivity, Chinese word segmentation, and delimiters. It does not expand the coverage of existing indexes. After you enable this feature, the system records token positions for existing tokenizable text indexes, including full-text indexes, text-type field indexes, and leaf fields indexed as text within JSON-type field indexes. Fields without a text index do not generate a position index.

For example, if the full-text index is disabled and only message has a text-type field index, and the error leaf field under the JSON field payload has a text-type index, only message and payload.error generate a position index.

Prerequisites

You have submitted a ticket to add the target Logstore to the position index allowlist.
The target Logstore has a full-text index or a text-type field index configured.

Enable a position index

Enable through the console

Log on to the Simple Log Service console.
Go to the query and analysis page of the target Logstore.
Choose Query and Analysis Properties > Properties.
Turn on the Enable Position Index switch at the bottom of the page, and then save the configuration.

The configuration takes effect in approximately one minute.

Enable through an SDK

The following example uses the Python SDK. Other language SDKs follow a similar approach.

Install the Python SDK:

pip install -U aliyun-log-python-sdk

Updating an index overwrites the entire configuration. Read the existing configuration first, then add the top-level field "position_index": true:

from aliyun.log import LogClient

class IndexConfigJson(dict):
    def to_json(self):
        return self

client = LogClient(
    "your-endpoint",
    "your-access-key-id",
    "your-access-key-secret",
)

project = "your-project"
logstore = "your-logstore"

config = client.get_index_config(project, logstore).get_body()
config.pop("lastModifyTime", None)
config["position_index"] = True
client.update_index(project, logstore, IndexConfigJson(config))

Use phrase queries

A phrase query uses # as an identifier and wraps the phrase in double quotation marks.

Query type	Syntax	Example
Full-text phrase query	`#"phrase"`	`#"connection reset"`
Field phrase query	`field:#"phrase"`	`message:#"connection reset"`

A phrase query matches tokenized token sequences and adjacent positions. It is not a character-by-character match. For example, connection reset by peer matches #"connection reset", but connection was reset by peer does not.

Capability comparison

A position index does not change the phrase query syntax, but enhances query capabilities:

Item	Without position index	With position index
Match method	Reads candidate logs and verifies them one by one	Performs exact match at the index level
Result pagination	Supports only sequential pagination	Supports sequential pagination and random page jumps
Histogram	Based on keyword candidate results	Based on precise phrase results
Condition combination	Does not support `NOT` or fuzzy query conditions	Supports `AND`, `OR`, `NOT`, and fuzzy query conditions
Analytic statement	Not supported	Supported
SPL	Not supported	Supported

The following examples demonstrate conditions and analytics that are available only with a position index enabled:

not #"connection reset"
#"connection reset" and host:api-*
#"connection reset" | SELECT level, count(*) GROUP BY level
#"connection reset" | where level = 'ERROR'

Usage notes

A position index takes effect only on newly written data or data with rebuilt indexes. Historical data does not automatically generate a position index.
The enhanced phrase query is used only when all data within the query range has a position index. If any data in the range lacks a position index, the query falls back to the non-position-index processing path.
Enabling a position index increases index construction and storage overhead. The increase primarily depends on the frequency of token occurrences.
High-frequency tokens and longer time ranges increase position data read volume. Narrow the time range, add field conditions, or use more specific phrases.
If a Logstore has both a full-text index and text-type field indexes with different tokenization configurations, the full-text phrase query #"phrase" matches only based on the full-text index tokenization result. For content that can be matched only through a field index, use the field phrase query field:#"phrase".
When you update an index through the API or SDK, preserve the existing index configuration to avoid overwriting and losing existing settings.