Deploy Prometheus on Kubernetes clusters
This topic describes how to deploy Prometheus on a Container Service for Kubernetes (ACK) cluster and store the collected monitoring data in a Time Series Database (TSDB) instance.
How Prometheus monitors Kubernetes
Prometheus is an open source monitoring tool for cloud-native applications. It identifies the servers to monitor using service discovery or static configuration. On Kubernetes, Prometheus can monitor:
System components built into the cluster, such as kube-apiserver, kube-controller-manager, and etcd
Static resource entities, such as node resource statuses and kernel events
Dynamic resource entities — abstract workloads such as Deployments, DaemonSets, and pods
Custom application objects, such as application-specific metrics and data
To monitor system components and static resource entities, specify monitoring methods in configuration files. To monitor dynamic resource entities, deploy Prometheus directly on the Kubernetes cluster.
Prerequisites
Before you begin, make sure you have:
An ACK cluster. See Create an ACK cluster
A TSDB instance. Note the instance connection URL from the TSDB console — you need it in step 2
Prometheus Operator deployed on the cluster. See Use the open source tool Prometheus to monitor a Kubernetes cluster
Deploy Prometheus and configure remote storage
Step 1: Deploy Prometheus on your ACK cluster
Use Prometheus Operator to deploy Prometheus. For detailed steps, see Use the open source tool Prometheus to monitor a Kubernetes cluster.
Step 2: Configure remote write and read
Edit the prometheus-prometheus.yaml file. Under the spec object, add the remoteWrite and remoteRead configuration items. Prometheus Operator uses camelCase for remote settings.
Replace ts-xxxxxxxxxxxx with your TSDB instance ID. To find your instance ID, go to the instance details page in the TSDB console.
The following example is based on a 3xlarge TSDB instance:
spec:
remoteWrite:
- url: "http://ts-xxxxxxxxxxxx.hitsdb.rds.aliyuncs.com:3242/api/prom_write"
queueConfig:
capacity: 10000
maxShards: 6
maxSamplesPerSend: 500
remoteRead:
- url: "http://ts-xxxxxxxxxxxx.hitsdb.rds.aliyuncs.com:3242/api/prom_read"
readRecent: trueThe queueConfig parameters control how Prometheus batches and sends metrics to TSDB:
| Parameter | Description | Value in example |
|---|---|---|
capacity | Buffer size of the queue. Increase this value if your cluster generates a high volume of metrics. | 10000 |
maxShards | Number of concurrent shards (parallel write connections) to TSDB. | 6 |
maxSamplesPerSend | Maximum number of samples sent in each request. | 500 |
Adjust these values based on your cluster's metric volume and TSDB instance type. For the full list of configurable fields in prometheus-prometheus.yaml, see the GitHub repository.
Step 3: Apply the configuration
Run the following command to apply the updated configuration:
kubectl apply -f manifests/prometheus-prometheus.yaml