Master PromQL with visual explanations

Lookback-based point selection

Time-series data is typically collected and reported at fixed intervals. However, PromQL queries can be executed at any time, so their time parameters often do not align perfectly with the data reporting timestamps. The PromQL evaluation process runs from the startTime to the endTime specified in the query. If no raw data point exists at a specific evaluation time T, PromQL looks back to find the most recent data point and uses it as the value for time T. At time T2, no valid data point was selected.

The following figure shows an end-to-end example of the point selection process for a single time series. This example uses the parameters start "10:00:00", end "10:11:30", and step "120s" and assumes that the original monitoring data is reported every 30s. The entire process involves six rounds of point selection. The six timestamps that are ultimately selected are "09:59:40", "10:01:40", "10:03:40", "10:05:40", "10:07:40", and "10:09:40". The PromQL-Engine then treats the data points at these six timestamps as the data points for "10:00:00", "10:02:00", "10:04:00", "10:06:00", "10:08:00", and "10:10:00" and uses them in the subsequent calculation process.

In the example above, the maximum lookback period for selecting a data point is controlled by the lookback-delta parameter, which defaults to 5 minutes in Prometheus and 3 minutes in MetricStore. For example, for the timestamp '10:00:00', the system looks back a maximum of 3 minutes. If no data point is found within the '09:57:00' to '10:00:00' time range, it means that no data for the '10:00:00' timestamp proceeds to the next stage of calculation. This selection method corresponds to the InstantVectorSelector in Prometheus, which is defined as VectorSelector in the Prometheus source code.

Window function point selection

As the name suggests, "window function point selection" means that this type of "point selection" logic exists only within the calculation process of window functions. For example, consider the same parameters as before, start "10:00:00", end "10:11:30", and step "120s", and assume that the window parameter for the window function is [5m], which represents 5 minutes.

Filter queries

This type of basic query statement does not involve any operators or functions. It supports adding filter conditions to any label by using only the =, !=, =~, or !~ matching operators. These filter conditions are typically pushed down to the storage layer, which means that the data is filtered before it enters the computation process.

The subsequent operations on the metric data are straightforward: they simply select data points based on the lookback-delta mechanism. Here are a few example queries for reference.

http_requests_total{replica="rep-a"}
http_requests_total{replica!~".*a"}
http_requests_total{environment=~"staging|testing|development",method!="GET"}

If you do not want to specify a particular metric and want to include the metric name in the fuzzy matching conditions, you can use the following examples.

• Note: Fuzzy matching on metric names is inefficient and generally not recommended for production use.

{__name__="http_requests_total", replica="rep-a"}
{__name__=~"http.*", replica="rep-a"}
{__name__=~".+", instance=~"127.*"}

Aggregation operators

• Note: The evaluation flow for all aggregation operations can be summarized as "first, select data points using the lookback-delta mechanism, and then aggregate them across time series."

PromQL currently supports the following aggregation operators: sum, avg, min, max, bottomk, topk, group, count, count_values, stddev, stdvar, quantile, limitk, and limit_ratio. This section uses the max and count operators and the following three time series as examples to describe the calculation process of aggregation operators.

timeseries 1: __name__="request_total_count", instance="127.0.0.1:10000", job="prometheus"

timeseries 2: __name__="request_total_count", instance="127.0.0.1:10001", job="vm-agent"

timeseries 3: __name__="request_total_count", instance="127.0.0.1:10002", job="vm-agent"

The query parameters are set as follows:

start: 10:00:00

end : 10:11:30

step: 120s

Functions

Functions in PromQL can be divided into two categories: window functions and non-window functions. For a detailed list of supported functions, see Prometheus Query functions. Functions that have an input parameter type of (v instant-vector) are non-window functions, and functions that have an input parameter type of (v range-vector) are window functions. Note that the primary difference between functions and aggregation operators is that aggregation operators perform aggregations across time series, whereas functions operate on the values or labels within a single time series.

This section continues to use the following three time series to explain how various functions work.

timeseries 1: __name__="request_total_count", instance="127.0.0.1:10000", job="prometheus"

timeseries 2: __name__="request_total_count", instance="127.0.0.1:10001", job="vm-agent"

timeseries 3: __name__="request_total_count", instance="127.0.0.1:10002", job="vm-agent"

The query parameters are set as follows:

start: 10:00:00

end : 10:11:30

step: 120s

label_replace(prometheus_build_info{}, "branch", "$1", "version", "(.)@.")

Binary expressions

PromQL supports three types of binary expressions: "Scalar <op> Scalar," "Vector <op> Scalar," and "Vector <op> Vector."

• Scalar <op> Scalar

This type is easy to understand: it performs a binary operation on two scalar values. Here are some examples:

1024 * 1024
9 / 3
3 ^ 2
3 == 1

• Vector <op> Scalar

The evaluation for this type of binary expression is very similar to that of a non-window function. It can be summarized as "first, select a data point using the lookback-delta mechanism, and then perform the corresponding binary operation on that data point." For a visual explanation, refer to the "Non-window functions" section. Here are a few examples:

request_total_count_min / 60
process_resident_memory_bytes / 1024 / 1024
query_latency_seconds * 1000

• Vector <op> Vector

This type of binary expression operates on two metric vectors. The overall process can be summarized as "first, select data points for both the left and right expressions using the lookback-delta mechanism, then applies the calculation to pairs of time series with identical labels. Unmatched time series are dropped from the result."

Metric: request_total_latency_ms

timeseries 1: __name__="request_total_latency_ms", instance="127.0.0.1:10000", job="prometheus"

timeseries 2: __name__="request_total_latency_ms", instance="127.0.0.1:10007", job="vm-agent"

timeseries 3: __name__="request_total_latency_ms", instance="127.0.0.1:10002", job="vm-agent"

Metric: request_total_count

timeseries 4: __name__="request_total_count", instance="127.0.0.1:10000", job="prometheus"

timeseries 5: __name__="request_total_count", instance="127.0.0.1:10001", job="vm-agent"

timeseries 6: __name__="request_total_count", instance="127.0.0.1:10002", job="vm-agent"

The query parameters are set as follows:

start: 10:00:00

end : 10:11:30

step: 120s

This section uses an example with data from two metrics and six time series to explain the calculation process for this type of binary expression. The PromQL query is query: request_total_latency_ms / request_total_count.

This Query can be interpreted as a calculation of the average latency of all requests at a specific time. First, data points are selected from the expressions on the left and right sides based on the 'lookback-delta' mechanism. This operation is identical to the lookback point selection process. Then, calculations are performed on the data from the time series on both sides that have identical labels. The following figure uses timeseries 1 and timeseries 4 as an example to illustrate the complete calculation process.

In the example above, only two pairs of time series have perfectly matching Labels: timeseries 1 and timeseries 4, and timeseries 3 and timeseries 6. Because timeseries 3 and timeseries 5 do not have a matching time series, they are skipped in subsequent calculations. Therefore, the final result set contains only two time series.

Vector matching

Vector matching is a core feature of PromQL syntax. Its structure is similar to a binary expression, but it allows for binary operations between two or more data vectors with different labels. Vector matching extends basic binary expressions by supporting the on and ignore operators to enable binary operations when labels do not match on both sides. The on operator matches only specific labels, whereas the ignore operator ignores specific labels. However, one-to-many, many-to-one, or even many-to-many relationships can exist between the labels on either side of an expression. The group_left and group_right operators are introduced to handle these scenarios.

Based on the matching results of labels from the left and right sides, there are three main scenarios: One-to-One, One-to-Many, and Many-to-One (Many-to-Many is not supported). This section uses the following six time series as an example to explain the calculation process for these three scenarios.

Metric: request_total_latency_ms

timeseries-1: __name__="request_total_latency_ms", instance="127.0.0.1:10000", job="prometheus", code="200" 90

timeseries-2: __name__="request_total_latency_ms", instance="127.0.0.1:10002", job="vm-agent", code="200" 20

timeseries-3: __name__="request_total_latency_ms", instance="127.0.0.1:10007", job="vm-agent", code="200" 60

Metric: request_total_count

timeseries-4: __name__="request_total_count", instance="127.0.0.1:10000", job="prometheus" 10

timeseries-5: __name__="request_total_count", instance="127.0.0.1:10002", job="vm-agent" 20

timeseries-6: __name__="request_total_count", instance="127.0.0.1:10007", job="sls-ilogtail" 30

One-to-One

This type of matching requires that the labels in the result sets of the left and right expressions have a one-to-one correspondence.

request_total_latency_ms / on(instance) request_total_count

--> Evaluation process:
timeseries-1 / timeseries-4  --Result--> instance="127.0.0.1:10000", 90/10
timeseries-2 / timeseries-5  --Result--> instance="127.0.0.1:10002", 20/20
timeseries-3 / timeseries-6  --Result--> instance="127.0.0.1:10007", 60/30

The example above is a query that uses the on operator. on(instance) specifies that only the "instance" label is considered when matching labels on both sides. Therefore, the query is also equivalent to request_total_latency_ms / ignore(job,code) request_total_count.

If you do not use the on operator, only the two time series pairs "timeseries 1 <--> timeseries 4" and "timeseries 2 <--> timeseries 5" in the sample data above can be matched for the subsequent binary operation.

• Many-to-One and One-to-Many

When you use the on and ignore operators, one-to-many or many-to-one matching scenarios can occur between the labels on each side. These scenarios result in an error by default. To handle this, you must use the group_left or group_right operator. group_left allows multiple time series from the left vector to match a single series from the right vector, while group_right allows multiple time series from the right vector to match a single series from the left vector.

request_total_latency_ms / on(job) group_left(instance, code) request_total_count
--> Evaluation process:
timeseries-1 / timeseries-4  --Result--> instance="127.0.0.1:10001",job="prometheus",code="200", 90/10
timeseries-2 / timeseries-5  --Result--> instance="127.0.0.1:10002",job="vm-agent",code="200", 20/20
timeseries-3 / timeseries-5  --Result--> instance="127.0.0.1:10007",job="vm-agent",code="200", 60/20

This example query uses the on operator with the group_left operator. When you use on(job), a many-to-one situation occurs, where both "timeseries-2" and "timeseries-3" on the left side can match "timeseries-5" on the right side. In this case, you must use group_left to allow this calculation. In addition, the group_left(instance, code) syntax specifies that the "instance" and "code" labels are preserved in the result set.

Other basic operations

max_over_time(sum(process_resident_memory_bytes) by (instance)[10m:1m])

Query expression breakdown:
sum(process_resident_memory_bytes) by (instance) --> step is adjusted to "1m"
max_over_time(xxxx[10m:1m])  --> window range is "10m", outer step is still "2m"

rate(sum(process_resident_memory_bytes) by (instance)[30m:30s])

Query expression breakdown:
sum(process_resident_memory_bytes) by (instance) --> step is adjusted to "30s"
rate(xxxx[30m:30s])  --> window range is "30m", outer step is still "2m"

Advanced usage examples

The previous sections explained the principles of basic operators, functions, and expressions in PromQL. In real-world business monitoring, nesting various operators is often necessary to express the intended calculation. The PromQL execution process is similar to the SQL Volcano Model. Leaf nodes read the data, and calculations are passed up layer by layer for each stage. This section uses two complex PromQL queries to describe the detailed expression structure and calculation process.

max ( max_over_time( process_memory_bytes [10m] ) / 1024 / 1024 ) by ( instance )

The syntax tree for this query is shown in the figure above. The execution process is broken down into the following four stages:

• Stage 1

expr_1: process_resident_memory_bytes{}[10m]

In this stage, the window function selects all data points within a time window for each round.

• Stage 2

expr_2: max_over_time(expr_1)

This stage calculates the maximum value of all data points within the time window for a single time series in each calculation round.

• Stage 3

expr_3: expr_2/1024/1024

After the first two stages, this stage performs the /1024/1024 mathematical operation on each numerical point in every time series of the result set.

• Stage 4

expr_4: max(expr_3) by (instance)

This stage performs a cross-time series aggregation on the result set from Stage 3. In each round, it calculates the maximum value for each instance group.

(sum(delta(container_network_receive_packets_dropped_total{namespace=~"kube-system"}[1m] offset 1h)) by (namespace,pod) 
/
sum(delta(container_network_receive_packets_total{namespace=~"kube-system"}[1m]  offset 1h)) by (namespace,pod)) > 0.02

The syntax tree for the query is shown in the figure above. The process is recursive and calls are made layer by layer. For a binary expression, the left subtree is executed before the right subtree. All queries can be divided into execution levels this way. The VectorSelector or MatrixSelector nodes read the data. Then, the expressions at each level are executed from the bottom up.

The StableNan identifier

Prometheus Engine defines a special float64 value called StableNan. This value identifies an invalid number. The identifier works as follows: If a data point with this value is selected based on the "lookback-delta" mechanism, the system considers that no valid point was selected in the current cycle. Note that a standard Math.NaN is a normal data point.

MetricStore supports writing StableNan values. The following example shows how to do this using the Go SDK for SLS:

var log = &sls.Log{
	Time: proto.Uint32(uint32(time.Now().Unix())),
	Contents: [ ]*sls.LogContent{
		{Key: proto.String("__name__"), Value: proto.String("test_metric")},
		{Key: proto.String("__labels__"), Value: proto.String("A#$#a|B#$#b")},
		{Key: proto.String("__time_nano__"), Value: proto.String("1687943952000000000")},
		// The fixed string "__STALE_NAN__" represents StableNan.
		{Key: proto.String("__value__"), Value: proto.String("__STALE_NAN__")},
	},
}

Other data types

• Exemplar

Exemplar is a feature that enhances the observability of monitoring systems. It lets you attach references or metadata for specific events to time series data points. Examples include distributed tracing IDs or log entry IDs. This mechanism helps you quickly correlate abnormal behavior in metrics with specific events. This greatly simplifies troubleshooting and performance analysis. Client libraries that support this feature typically generate and attach Exemplars in the application. Prometheus then collects them during data scraping.

In distributed systems, you can integrate Exemplars with distributed tracing systems, such as Jaeger or Zipkin. This provides a direct link from metric anomalies to specific traces, making problem diagnosis more efficient and accurate. Grafana supports this feature. It lets you visually inspect and analyze these event correlations on metric graphs. This enhances the functionality and user experience of the entire monitoring system. Overall, Exemplars provide important visualization and analysis capabilities for monitoring and troubleshooting complex systems.

• NativeHistogram

In Prometheus, the traditional Histogram model relies on user-defined, fixed bucket boundaries. It uses multiple metrics, such as _bucket, _sum, and _count, to record data distribution. This method can lead to inaccurate distribution representations. This is especially true when you are unsure of the actual data distribution and find it difficult to define appropriate buckets. In addition, using multiple metrics increases the complexity of data storage and queries. You must process these separate metrics to get complete distribution information.

In contrast, the NativeHistogram model uses an adaptive bucket partitioning algorithm. It dynamically adjusts buckets to fit the actual data distribution. You no longer need to manually define bucket boundaries. It merges the multiple metrics into a single data structure. This significantly reduces the data volume and query complexity of monitoring metrics. Data is no longer represented as simple floating-point numbers. Instead, it is stored as a more complex byte array. This allows you to directly save a snapshot of the entire distribution at a specific point in time. NativeHistogram provides higher accuracy and efficiency. It simplifies the use and management of monitoring systems. It is especially useful for handling complex and highly variable data distribution scenarios.

Why does MetricStore not yet support these data types?

1. Exemplar

Essentially, Prometheus lacks general-purpose storage capabilities. This is why the special Exemplar data type was designed. Its query interface is similar to the PromQL query interface. It also reads the entire Exemplar data based on the metric name and label conditions. The general-purpose storage capabilities of Log Service (SLS) Logstores are a perfect fit for this scenario and offer higher storage and query efficiency.

2. NativeHistogram

This feature has been experimental since its introduction in Prometheus v2.40.0. You must enable it using --enable-feature=native-histograms. Each release may introduce breaking changes, leading to unstable APIs and storage formats. In addition, the supporting tool ecosystem is insufficient. For example, components such as Exporter and AlertManager do not support it. Derived storage products, such as Mimir and VictoriaMetrics, also do not support this feature. MetricStore will consider gradually adopting this feature after its API becomes stable, its performance is optimized, and its toolchain matures.

Query APIs

Prometheus supports the /query and /query_range HTTP APIs to query and analyze time series data.

The /query_range API performs range queries. It accepts a PromQL statement along with start time, end time, and step parameters. The API returns the data series within the specified time range. All execution processes described earlier fall under the /query_range API. This API is suitable for analyzing historical data and observing trends, such as generating charts to view resource usage over time.

The /query API primarily performs instant queries, returning metric values at a specific point in time. Simply put, the /query execution process is a subset of the /query_range process. /query performs only a single calculation at a specified time. This API is commonly used for real-time alerting.

Metadata APIs

Prometheus also supports metadata query APIs. These APIs retrieve label and metric information related to time series without fetching specific timestamps and numerical data. The /labels API provides a list of all label names in the current storage. The /label/<label_name>/values API provides all possible values for a given label name. The /series API returns time series metadata that matches specific label conditions. These APIs provide a convenient way to explore and understand the structure of Prometheus data by retrieving metrics, labels, and label values for a time period without fetching the raw data points.

Front-end tools such as Grafana mainly use metadata APIs to implement features like PromQL auto-completion and dynamic configuration. If you call the API without a proper match[] parameter to limit the query scope, the storage layer might process millions of time series to build the result. This can increase query time from hundreds of milliseconds to tens of seconds, and raise memory usage to the gigabyte level. In large-scale production environments, an improper match[] parameter can even cause query timeouts or make the service unavailable. Always use the match[]=<metric_selector> parameter to restrict the query to specific time series.

For more information about the HTTP APIs and error codes that MetricStore currently supports, see MetricStore HTTP API details and MetricStore HTTP API return values.