PromQL and SQL query Metricstore common cases

Query single metric data

process_resident_memory_bytes reports resident memory for a process. The following query returns all time series for this metric across the selected time range.

process_resident_memory_bytes

Filter metric data by label

Add label selectors inside {} to scope results to a specific label value. The following query returns resident memory only for the sls-mall cluster, then converts bytes to MB.

process_resident_memory_bytes{cluster="sls-mall"}/1024/1024

Use regular expressions for label filtering

Use =~ instead of = to match label values with a regular expression. The following query returns data where the cluster is sls-mall or sls-demo.

process_resident_memory_bytes{cluster=~"sls-mall|sls-demo"}

Calculate the maximum value of a metric

max by (cluster) collapses all time series within each cluster into a single series, keeping only the highest value at each timestamp. The by (cluster) clause preserves the cluster label and removes all others. The following query returns the peak heap memory per cluster in MB.

max by (cluster) (go_memstats_heap_inuse_bytes) / 1024 / 1024

Calculate the number of time series for a metric

count() returns the number of active time series that match a selector. Use this to check how many instances are reporting a given metric. The following query counts all time series for apiserver_request_total in the sls-mall cluster.

count(apiserver_request_total{cluster="sls-mall"})

Calculate a metric's rate of change over time

rate() calculates the per-second average rate of change over a time window. It is designed for counter metrics that only increase — use it to compute CPU utilization, request throughput, and similar rates. The following query returns the per-second CPU usage rate over the last minute.

rate(process_cpu_seconds_total[1m])

Calculate the difference of a metric from n minutes ago

delta() calculates the difference between the first and last value of a metric within a time window. Unlike rate(), delta() works with gauges — metrics that can increase or decrease. The following query returns the change in goroutine count over the last minute.

delta(go_goroutines[1m])

Nested use of multiple operators and functions

PromQL operators and functions can be nested. Inner expressions are evaluated first and passed as input to the outer function. The following query first calculates the maximum request count per cluster and resource, then selects the top 3 combinations.

topk(3, max by (cluster, resource)(apiserver_request_total))

Calculations between multiple values

PromQL supports standard arithmetic operators (+, -, *, /, %) on plain numbers. The following example evaluates to 3.

1 + 2

Calculations between metrics and values

Arithmetic operators also apply between a metric and a scalar. The following query converts the process_resident_memory_bytes metric from bytes to MB.

process_resident_memory_bytes / 1024 / 1024

Binary operations between multiple metrics

PromQL supports binary operations between two metrics. Matching is performed on label sets — only time series with identical labels in both metrics are paired. The following query calculates the ratio of ready DaemonSet pods to scheduled pods.

kube_daemonset_status_number_ready{job="kube-state-metrics"} 
/ 
kube_daemonset_status_desired_number_scheduled{job="kube-state-metrics"}

How to correctly use subquery

Functions such as rate(), delta(), increase(), and {agg}_over_time() operate only on raw metric data — they cannot take the output of an aggregation operator as input. Use a subquery to work around this constraint.

The [a:b] syntax in a subquery specifies the range (a) and step (b).

• Supported — max_over_time applied to a raw metric:

  max_over_time(go_goroutines[1m]) # Returns the maximum goroutine count in the previous minute, per time series.

• Not supported — max_over_time applied to an aggregated result:

  max_over_time(max(go_goroutines)[1m]) # Invalid: max(go_goroutines) produces a scalar, not a raw metric.

• Correct usage — use a subquery to apply the range function after aggregation:

  # First aggregate across time series, then apply the range function via subquery.
  max_over_time(max(go_goroutines)[1m:10s])
  # [1m:10s] — evaluate max(go_goroutines) at 10-second steps over the past 1 minute, then take the max of those values.

Query all raw data under a metric

Return all time series and samples for process_resident_memory_bytes within the selected time range. (Demo )

 *| SELECT * FROM "metrics_store_name.prom" WHERE __name__ = 'process_resident_memory_bytes'

Query raw data with a specific instance value for a metric

Extend the previous query with element_at() to filter by a specific label value. The following query returns all samples where the metric is process_resident_memory_bytes and the instance is 172.20.0.143:8084. (Demo)

*| SELECT * FROM "metrics_store_name.prom" WHERE __name__ = 'process_resident_memory_bytes' and element_at(__labels__, 'instance')='172.20.0.143:8084' limit all

Aggregate time series data by minute

Align timestamps to minute boundaries and calculate the maximum metric value per instance. (Demo)

*| SELECT (__time_nano__ - __time_nano__ % 60000000)/1000000.0 as t, element_at(__labels__, 'instance') as instance, max(__value__) as val FROM "metrics_store_name.prom" WHERE __name__ = 'process_resident_memory_bytes' group by t, instance limit all