Deliver Prometheus data to ApsaraMQ for Kafka-Managed Service for Prometheus(Prometheus)-阿里云帮助中心

Prerequisites

Ensure that a Prometheus instance is connected. For more information, see the following topics:
Deploy an ApsaraMQ for Kafka instance and create the necessary resources, such as a topic. For more information, see Overview.
EventBridge has been activated. For more information, see Activate EventBridge and grant permissions.

Limits

If the Prometheus instance and the destination reside in different VPCs, add the destination vSwitch CIDR block to the Prometheus instance allowlist to avoid network connection failures.

Supported data sources:

Instance type	Description
Prometheus for Cloud Services	Excludes free instances whose names start with cloud-product-prometheus.
Prometheus for Container Service	None
Prometheus for Flink Serverless	None
Prometheus for Kubernetes	None
General-purpose	Excludes data reported through OpenTelemetry endpoints.

Only real-time data generated after task creation is shipped; historical data is not supported.

Step 1: Create a data delivery task

Log on to the Cloud Monitor console.
In the left-side navigation pane, click Prometheus Monitoring > Data Shipping.
On the Data Shipping page, select a destination region in the top menu bar and click Create Task.
In the dialog box that appears, enter a Task Name and Task description, and click OK.

On the Edit Task page, configure the data source and destination.

Click + Add Data Source, configure the following parameters, and then click OK.

Parameter	Description	Example
Prometheus Instance	The source Prometheus instance.	c78cb8273c02*****
Data Filtering	Filter data by metric labels. Supports regular expressions. Separate multiple conditions with line breaks (AND logic).	`__name__=AliyunEcs_CPUUtilization\|AliyunEcs_memory_usedutilization regionId=cn-hangzhou id=i-2ze0mxp.*`
Data Labeling	Additional labels for shipped metric data. Separate with line breaks.	`deliver_test_key1=ssss deliver_test_key2=yyyy`

Click Add Target, set Target Type to ApsaraMQ for Kafka, configure the remaining parameters, and then click OK.

On the Edit Task page, click OK, and then click Save to create the data shipping task.

Step 2: View Prometheus monitoring data

Note

To reduce the load on the target, the service delivers Prometheus monitoring data to Kafka as a JSON array compressed by using Snappy. For more information, see Snappy compression format.

View in console

Log on to the ApsaraMQ for Kafka console.
On the Overview page, select a region in the Resource Distribution section.
On the Instances page, click the name of the target instance.
In the navigation pane on the left, click Topics. Find the target topic and click Details in the Actions column. You can then view the imported data on the CloudMonitor or Message Query tab.

View with client

Initialize a Kafka client. For more information, see Use a single consumer to subscribe to messages.

Add the following code to the KafkaConsumerDemo.java file. The code consumes data and decompresses it by using Snappy after the Kafka client is initialized:

public static void main(String[] args) {

        // Initialize the Kafka consumer first.
        
        while (true){
            try {
                ConsumerRecords<String, byte[]> records = consumer.poll(1000);
                // You must consume the records before the next poll, and the total time must not exceed SESSION_TIMEOUT_MS_CONFIG.
                // We recommend that you use a dedicated thread pool to consume messages and return results asynchronously.
                for (ConsumerRecord<String, byte[]> record : records) {
                    byte[] compressedData = record.value();
                    byte[] data = Snappy.uncompress(compressedData);
                    
                    System.out.println(new String(data));
                }
            } catch (Exception e) {
                try {
                    Thread.sleep(1000);
                } catch (Throwable ignore) {

                }

                e.printStackTrace();
            }
        }
}

Compile and run the KafkaConsumerDemo.java file. The metric data is output in the following JSON format.

[{
	"instance": "*****",
	"pod": "*****",
	"rejected": "false",
	"type": "validate",
	"pod_name": "*****",
	"endpoint": "http-metrics",
	"__name__": "apiserver_admission_controller_admission_duration_seconds_bucket",
	"service": "kubernetes",
	"name": "*****",
	"namespace": "default",
	"le": "2.5",
	"job": "apiserver",
	"operation": "UPDATE",
	"value": "675.0",
	"timestamp": "1698732988354"
}, {
	"instance": "*****",
	"pod": "*****",
	"rejected": "false",
	"type": "validate",
	"pod_name": "*****",
	"endpoint": "http-metrics",
	"__name__": "apiserver_admission_controller_admission_duration_seconds_bucket",
	"service": "kubernetes",
	"name": "*****",
	"namespace": "default",
	"le": "+Inf",
	"job": "apiserver",
	"operation": "UPDATE",
	"value": "675.0",
	"timestamp": "1698732988354"
}, {
	"instance": "*****",
	"pod": "*****",
	"rejected": "false",
	"type": "validate",
	"pod_name": "*****",
	"endpoint": "http-metrics",
	"__name__": "apiserver_admission_controller_admission_duration_seconds_bucket",
	"service": "kubernetes",
	"name": "*****",
	"namespace": "default",
	"le": "0.005",
	"job": "apiserver",
	"operation": "UPDATE",
	"value": "1037.0",
	"timestamp": "1698732988519"
}, {
	"instance": "*****",
	"pod": "*****",
	"rejected": "false",
	"type": "validate",
	"pod_name": "*****",
	"endpoint": "http-metrics",
	"__name__": "apiserver_admission_controller_admission_duration_seconds_bucket",
	"service": "kubernetes",
	"name": "*****",
	"namespace": "default",
	"le": "0.025",
	"job": "apiserver",
	"operation": "UPDATE",
	"value": "1037.0",
	"timestamp": "1698732988519"
}]