TimeStream集成Prometheus接口

阿里云Elasticsearch团队自研的TimeStream时序引擎增强插件,支持使用PromQL查询Elasticsearch数据,无缝对接Prometheus+Grafana。本文介绍如何基于Elasticsearch TimeStream使用Prometheus集成接口。

背景信息

TimeStream是阿里云Elasticsearch团队自研,并结合Elastic社区时序类产品特性共建的时序引擎。阿里云Elasticsearch提供的TimeStream时序增强功能插件,优化了Elasticsearch在存储指标数据方面的DSL(Domain-Specific Language)查询复杂且慢以及存储成本过高等问题。详细信息请参见TimeStream时序增强引擎介绍

前提条件

已创建阿里云Elasticsearch实例,且实例版本为通用商业版7.16及以上、内核版本为1.7.0及以上,或者实例版本为通用商业版7.10、内核版本为1.8.0及以上。具体操作请参见创建阿里云Elasticsearch实例

Prometheus API说明

Prometheus API除了可以访问通过Prometheus Remote Write方式写入的数据,也可以访问任意TimeStream索引。

Prometheus API支持去掉返回的指标和维度的前后缀关键字。如果使用默认的数据模型,指标前缀metric.会被去掉,维度前缀label.会被去掉。

例如,通过Prometheus Remote Write方式写入如下数据:
{
  "labels": {
    "instance": "127.0.0.1:9114",
    "job": "elasticsearch"
  },
  "metrics": {
    "up": 0.0,
    "scrape_samples_post_metric_relabeling": 0.0,
    "scrape_samples_scraped": 0.0,
    "scrape_duration_seconds": 4.45999E-4,
    "scrape_series_added": 0.0
  },
  "@timestamp": "1655717638795"
}
Prometheus API返回的维度字段会去掉前缀labels.,/api/v1/labels接口返回结果如下:
{
  "status" : "success",
  "data" : [
    "__name__",
    "instance",
    "job"
  ]
}
                
Prometheus API返回的指标字段会去掉前缀metrics.,/api/v1/metadata接口返回结果如下:
{
  "status" : "success",
  "data" : {
    "scrape_samples_post_metric_relabeling" : [
      {
        "type" : "gauge",
        "help" : "",
        "unit" : ""
      }
    ],
    "scrape_samples_scraped" : [
      {
        "type" : "gauge",
        "help" : "",
        "unit" : ""
      }
    ],
    "scrape_duration_seconds" : [
      {
        "type" : "gauge",
        "help" : "",
        "unit" : ""
      }
    ],
    "scrape_series_added" : [
      {
        "type" : "gauge",
        "help" : "",
        "unit" : ""
      }
    ],
    "up" : [
      {
        "type" : "gauge",
        "help" : "",
        "unit" : ""
      }
    ]
  }
}
如果您自定义了数据模型,则需要额外配置前后缀关键字,否则Prometheus API返回的就是实际的数据。配置前后缀关键字的示例如下:
PUT _time_stream/{name}
{
  "time_stream": {
    "labels_fields": "@labels.*_l",
    "metrics_fields": "@metrics.*_m",
    "label_prefix": "@labels.",
    "label_suffix": "_l",
    "metric_prefix": "@metrics.",
    "metric_suffix": "_m"
  }
}

RemoteWrite接口

API

POST /_time_stream/prom_write/{index}
---PB data---

使用说明

在Prometheus中配置Remote Write,即可将Prometheus数据同步到Elasticsearch,配置示例如下:
remote_write:
  - url: "http://127.0.0.1:9200/_time_stream/prom_write/prom_index"
    basic_auth:
     username: elastic
     password: xxxx

Prometheus通过Remote Write写入数据到TimeStream索引,只支持默认的时序数据模型,即维度前缀是label.,指标前缀是metric.

Instant Query接口

API

GET  /_time_stream/prom/{index}/query
POST /_time_stream/prom/{index}/query

使用说明

Prometheus Instant Query接口,详细信息请参见Prometheus instant queries

Range Query接口

API

GET  /_time_stream/prom/{index}/query_range
POST /_time_stream/prom/{index}/query_range

使用说明

Prometheus Range Query接口,详细信息请参见Prometheus range queries

Series Metadata接口

API

GET  /_time_stream/prom/{index}/series
POST /_time_stream/prom/{index}/series

使用说明

Prometheus查询时间线的接口,详细信息请参见Prometheus finding series by label matchers

Labels Metadata接口

API

GET  /_time_stream/prom/{index}/labels
POST /_time_stream/prom/{index}/labels

使用说明

Prometheus查询维度(label)列表的接口,详细信息请参见Prometheus getting label names

Label Values Metadata接口

API

GET /_time_stream/prom/{index}/label/<label_name>/values

使用说明

Prometheus查询维度对应values列表的接口,详细信息请参见Prometheus querying label values

Metric Metadata接口

API

GET /_time_stream/prom/{index}/metadata

使用说明

Prometheus查询指标(Metrics)列表的接口,详细信息请参见Prometheus querying metric metadata

TimeStream PromQL支持情况

表达式支持情况

表达式类型是否支持说明
powOpbinaryOperator幂乘
unaryOpbinaryOperator正、负
multOpbinaryOperator乘、除、取模
addOpbinaryOperator加、减
compareOpbinaryOperator比较(==,>=, >, <=, <, !=)
andUnlessOpbinaryOperator集合运算(and、or、unless)
orOpbinaryOperator集合运算(or)
functionvector函数
aggregationvector聚合运算
instantSelectorvector瞬时选择器
matrixSelectorvector向量选择器
offsetvector偏移量
literalvector常量
labelMatchervector标签匹配
groupinggroupgroup聚合,包括on、ignoring、group_left、group_right

Aggregation operator支持情况

Aggregation operator是否支持
sum(calculate sum over dimensions)
min(select minimum over dimensions)
max(select maximum over dimensions)
avg(calculate the average over dimensions)
group(all values in the resulting vector are 1)
stddev(calculate population standard deviation over dimensions)
stdvar(calculate population standard variance over dimensions)
count(count number of elements in the vector)
count_values(count number of elements with the same value)
bottomk(smallest k elements by sample value)
topk(largest k elements by sample value)
quantile(calculate φ-quantile (0 ≤ φ ≤ 1) over dimensions)

Function支持情况

Function是否支持
abs(v instant-vector)
absent(v instant-vector)
absent_over_time(v range-vector)
ceil(v instant-vector)
changes(v range-vector)
clamp(v instant-vector, min scalar, max scalar)
clamp_max(v instant-vector, max scalar)
clamp_min(v instant-vector, min scalar)
day_of_month(v=vector(time()) instant-vector)
day_of_week(v=vector(time()) instant-vector)
days_in_month(v=vector(time()) instant-vector)
delta(v range-vector)
deriv(v range-vector)
exp(v instant-vector)
floor(v instant-vector)
histogram_quantile(φ scalar, b instant-vector)
holt_winters(v range-vector, sf scalar, tf scalar)
hour(v=vector(time()) instant-vector)
idelta(v range-vector)
increase(v range-vector)
irate(v range-vector)
label_join(v instant-vector, dst_label string, separator string, src_label_1 string, src_label_2 string, ...)
label_replace(v instant-vector, dst_label string, replacement string, src_label string, regex string)
ln(v instant-vector)
log2(v instant-vector)
log10(v instant-vector)
minute(v=vector(time()) instant-vector)
month(v=vector(time()) instant-vector)
predict_linear(v range-vector, t scalar)
rate(v range-vector)
resets(v range-vector)
round(v instant-vector, to_nearest=1 scalar)
scalar(v instant-vector)
sgn(v instant-vector)
sort(v instant-vector)
sort_desc()
sqrt(v instant-vector)
time()
timestamp(v instant-vector)
vector(s scalar)
year(v=vector(time()) instant-vector)
avg_over_time(range-vector)
min_over_time(range-vector)
max_over_time(range-vector)
sum_over_time(range-vector)
count_over_time(range-vector)
quantile_over_time(scalar, range-vector)
stddev_over_time(range-vector)
stdvar_over_time(range-vector)
last_over_time(range-vector)
present_over_time(range-vector)
acos(v instant-vector)
acosh(v instant-vector)
asin(v instant-vector)
asinh(v instant-vector)
atan(v instant-vector)
atanh(v instant-vector)
cos(v instant-vector)
cosh(v instant-vector)
sin(v instant-vector)
sinh(v instant-vector)
tan(v instant-vector)
tanh(v instant-vector)
deg(v instant-vector)
pi()
rad(v instant-vector)