FAQ about metric data queries

Why does rate()/increase() produce abnormal values?

The calculation logic in the Prometheus Engine source code causes this behavior. For details, see the source code.

The following code snippet shows how rate() and increase() are calculated within a time window. The difference between the last and first values is assigned to resultFloat, and then the remaining data points are iterated. If a value decrease is detected, the previous value (prevValue) is added to resultFloat, which causes the final result to become abnormally large.

switch {
	case len(samples.Histograms) > 1:
		numSamplesMinusOne = len(samples.Histograms) – 1
		firstT = samples.Histograms[0].T
		lastT = samples.Histograms[numSamplesMinusOne].T
		var newAnnos annotations.Annotations
		resultHistogram, newAnnos = histogramRate(samples.Histograms, isCounter, metricName, args[0].PositionRange())
		annos.Merge(newAnnos)
		if resultHistogram == nil {
			// The histograms are not compatible with each other.
			return enh.Out, annos
		}
	case len(samples.Floats) > 1:
		numSamplesMinusOne = len(samples.Floats) – 1
		firstT = samples.Floats[0].T
		lastT = samples.Floats[numSamplesMinusOne].T
		resultFloat = samples.Floats[numSamplesMinusOne].F – samples.Floats[0].F
		if !isCounter {
			break
		}
		// Handle counter resets:
		prevValue := samples.Floats[0].F
		for _, currPoint := range samples.Floats[1:] {
			if currPoint.F < prevValue {
				resultFloat += prevValue
			}
			prevValue = currPoint.F
		}
	default:
		return enh.Out, annos
	}
				resultFloat += prevValue
			}
			prevValue = currPoint.F
		}
	default:
		// TODO: add RangeTooShortWarning
		return enh.Out, annos
	}

Troubleshooting

1. Use a PromQL query to determine if a value decrease has occurred. First, narrow the time filter to the period when the anomaly occurred, and then run the following PromQL query. Replace xxxx_metric with your actual metric name. If the query returns any data points, a value decrease occurred.

   Query: (xxxx_metric{} - xxxx_metric{} offset 1s) < 0
   Step parameter: 1s

2. Use an SQL query to retrieve the original anomalous data points for the affected time series. For example:

   * | select *, from_unixtime(__time_nano__/1000000.0)  from "Your_Metricstore_Name.prom" where __name__='your_metric_name' and element_at(__labels__, 'your_label_key_for_filtering')='your_label_value_for_filtering' order by __time_nano__ 
   // We recommend that you add multiple element_at functions to the SQL query to further narrow down the time series.

3. The SQL result set is sorted by timestamp, allowing you to directly observe the raw, anomalous data points that were written.

Why does PromQL return no data?

First, check the following two areas.

If neither scenario applies, the problem is likely caused by the vector selection logic and the lookback-delta mechanism of the Prometheus compute engine.

RangeVectorSelector:
rate(http_requests_total[5m])
delta(http_requests_total[5m])
count_over_time(http_requests_total[5m])
InstantVector:
http_requests_total
absent(http_requests_total)
count(http_requests_total)

Solutions

1. Retrieve data for a specific write time point by adjusting the startTime, endTime, and step query parameters to align with that time point.

   For example, the following query returns seven data points. However, metric data is often collected at irregular intervals in practice, making it impractical to manually align query parameters. This solution is not recommended for general use.

   startTime: 10:00:00
   endTime  : 18:00:00
   step     : 1h
   query    : count(up)

2. Decrease the step parameter. For example, setting the step to 3 minutes in the InstantVectorSelector scenario makes the selection process more frequent, increasing the likelihood of finding sparsely distributed raw data points.

3. Increase the lookback-delta parameter or the time range in the [xx] operator. This expands the time window for vector selection, making it more likely to include relevant data points.

Why PromQL returns data after writes stop?

The lookback-delta mechanism in the Prometheus compute engine causes this behavior. For details, see lookback-delta mechanism.

Why query timestamps differ from write times?

The lookback-delta mechanism in the Prometheus compute engine causes this behavior. For details, see lookback-delta mechanism.

Why repeated queries return different results?

Two possible reasons exist:

1. If you are using a relative time range, the absolute start and end times of your query change with each execution. As a result, the lookback-delta operation may select different data points each time, leading to different results. This is expected behavior and consistent with the Prometheus standard.

2. If there is a long interval between two executions for the same fixed time range, the first query might have run before all the data for that period arrived at Simple Log Service due to ingestion delays. The difference in available data between queries leads to inconsistent results. To mitigate this, use an offset modifier in your PromQL query to shift the query range back in time, which helps avoid errors caused by ingestion latency.

Query fails due to resource limits

If your query returns an error containing text such as "too many time series or items, xxxxxx" or "too many time series or items in parallelMaster node, xxxxx", the query attempted to read a large volume of data and triggered a memory limit at the compute layer.

Solutions

1. Reduce the query time range.

2. If you must run the query over the specified time range, enable concurrent computing. For more information, see the concurrent computing configurations.

Vector cannot contain metrics with same labelset

• Reason 1: The LabelName entries in the __labels__ field value are not in alphabetical order.

  The __labels__ field in MetricStore consists of multiple labels in the LabelName#$#LabelValue format, separated by vertical bars (|). All labels must be sorted alphabetically by LabelName. For details, see metric identifier.

  This error occurs when the label names in the __labels__ field value are not in alphabetical order. Use the following SQL statement to verify whether they are sorted correctly.

  * | select * from (
  select __labels__, array_join(array_sort(split(__labels__, '|')), '|') as rightLabels from "Your_Metricstore_Name.prom" where __name__!=''
  ) where __labels__ != rightLabels

• Reason 2: The __labels__ field contains an empty LabelValue.

  In Prometheus Engine, labels with an empty LabelValue are invalid. Prometheus Engine removes these labels during computation, which causes this error. Follow these steps to check whether the __labels__ field contains a label with an empty LabelValue.

  1. Narrow the query time range to locate the approximate time when the error occurs.

  2. Run the following SQL query.

     * | select __labels__ from "Your_Metricstore_Name.prom" where __name__!='' and regexp_like(__labels__, '.*#\$#\|.*|.*#\$#$')

     If the query returns any results, it confirms the presence of empty label values. Modify the code at your data reporting source to remove these invalid labels.

Why are labels missing from query results?

This issue occurs when the LabelName entries in the __labels__ field value are not sorted alphabetically. Use the following SQL statement to verify the sort order.

* | select * from (
select __labels__, array_join(array_sort(split(__labels__, '|')), '|') as rightLabels from "Your_Metricstore_Name.prom" where __name__!=''
) where __labels__ != rightLabels

Why by/without clauses give unexpected results?

This problem occurs when the label names in the __labels__ field value are not sorted alphabetically. Use the following SQL statement to verify the sort order.

* | select * from (
select __labels__, array_join(array_sort(split(__labels__, '|')), '|') as rightLabels from "Your_Metricstore_Name.prom" where __name__!=''
) where __labels__ != rightLabels

No data is displayed in metric exploration

No data may exist within the default query time range.

When you select a time range for your query, ensure that data exists within that range.

Why are time series missing from results?

When you run a query on the query and analysis page for a Metricstore in the Simple Log Service console, the limit parameter restricts the number of results returned by SQL or PromQL queries. Increase this value as needed.

Warnings in PromQL results

A possible cause is that the system could not read all data within the time range because the read capacity of a shard was reached. To resolve this, either narrow the query time range or split the shard to increase overall throughput.

Error: Maximum resolution exceeded

In Prometheus, each time series is limited to a maximum of 11,000 data points, meaning (endTime - startTime) / step cannot exceed 11,000. Narrow the query time range or increase the step parameter.