Model Prometheus metric sets

Definition and purpose of MetricSet

A MetricSet serves as the core component for metric modeling and has the following key responsibilities:

• Metric organization: Organizes related metrics into logical collections for easier management and querying.

• Label management: Defines common labels and filter rules for metrics.

• Query optimization: Provides efficient metric query capabilities through a unified query interface.

• Semantic expression: Provides rich metadata and multi-language support for metrics.

MetricSet structure overview

Observable Data System
├── EntitySet                     # EntitySet
├── TelemetryDataSet              # TelemetryDataSet (Metrics, Logs, Traces)
    └── MetricSet                 # MetricSet
        ├── labels                # Label definition
        │   ├── keys: Field[]     # Label field list
        │   ├── dynamic: boolean  # Dynamic label generation
        │   └── filter: string    # Label filter
        └── metrics: Metric[]     # Metric list
            ├── name              # Metric name
            ├── generator         # Query generator
            ├── aggregator        # Aggregation method
            └── data_format       # Format method

Label design principles

Labels define the dimensional properties of metrics. Follow these design principles:

• Prioritize generality: Labels at the MetricSet level should be dimensions common to all metrics.

• Generate dynamically: Dynamically generate labels to avoid hard coding.

• Filter efficiently: Labels should support efficient indexing and filter queries.

• Control cardinality: Avoid high-cardinality labels that can cause performance issues.

Label property configuration

Label field definition

Each label field inherits all properties from Field. Pay attention to the following attributes:

Label configuration examples

Label configuration for a Kubernetes scenario

labels:
  dynamic: true
  filter: 'kube_deployment_spec_replicas'
  keys:
    - name: namespace
      display_name:
        en_us: Namespace
      type: string
      filterable: true
      analysable: true
      pattern: ".*"
    - name: deployment  
      display_name:
        en_us: Deployment
      type: string
      filterable: true
      analysable: true
      pattern: ".*"

Label configuration for an Application Performance Monitoring (APM) scenario

labels:
  dynamic: true
  filter: 'arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc{callKind=~"http|rpc|custom_entry|server|consumer|schedule"}'
  keys:
    - name: service
      display_name:
        zh_cn: Service Name
        en_us: Service
      type: string
      filterable: true
      analysable: true
    - name: rpc
      display_name:
        zh_cn: Operation Name  
        en_us: Operation
      type: string
      filterable: true
      analysable: true

Metric design principles

Metrics are the core components of a MetricSet. Each metric represents a queryable monitoring dimension:

• Business semantics: The metric name should clearly reflect its business meaning.

• Complete calculation: The `generator` should contain the complete calculation logic, not just the raw metric.

• Clear units: Configure the `data_format` and `unit` properties correctly.

• Appropriate aggregation: Select the correct `aggregator` based on the metric's attributes.

Core metric properties

Based on the field system, a Metric extends the following monitoring-specific properties:

Data formatting specification

Metrics are often used to create charts, so data format fields matter. Whenever possible, use the enumeration values defined in the Schema instead of custom formats.

For more information about formatting options, see the Field system topic.

Aggregator selection guide

Choosing the correct aggregator is essential for accurate metric queries:

Note: For metrics whose ratio is already calculated in the `generator` (such as `usage/limit * 100`), do not set an `aggregator`. The UModel system automatically selects an aggregation policy as needed.

The following table shows common combinations of `generator` and `aggregator`.

- name: request_count
  generator: 'sum_over_time_lcro(arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc{callKind=~"http|rpc|custom_entry|server|consumer|schedule"}[60s])'
  aggregator: sum
  data_format: KMB

2. Replica count metric: Desired number of deployment replicas.

- name: deployment_desired_replicas
  generator: kube_deployment_spec_replicas{}
  aggregator: sum
  data_format: KMB

3. Ratio metric without a set aggregator: Deployment availability.

- name: deployment_availability_rate
  generator: (sum by (namespace, deployment) (kube_deployment_status_replicas_ready{}) / (sum by (namespace, deployment) (kube_deployment_spec_replicas{}))!=0) * 100
  # Do not set an aggregator
  data_format: percent

4. Average latency metric: The average response time from APM.

- name: avg_request_latency_seconds
  generator: 'sum(sum_over_time_lcro(arms_app_requests_seconds_ign_destid_endpoint_parent_ppid_prpc{callKind=~"http|rpc|custom_entry|server|consumer|schedule"}[60s])) / sum(sum_over_time_lcro(arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc{callKind=~"http|rpc|custom_entry|server|consumer|schedule"}[60s]))'
  # Do not set an aggregator
  data_format: s

Golden metric identifier

Golden metrics are the most critical monitoring metrics:

• Quantity limit: You can set 3 to 5 golden metrics for each MetricSet, but do not exceed 8.

• Selection criteria: Core metrics that directly reflect the health of the system.

• Scenarios: Alert rules, dashboards, and automated Operations and Maintenance (O&M).

Metric configuration examples

• CPU usage metric.

  - name: deployment_cpu_usage_vs_requests
    display_name:
      zh_cn: CPU Usage vs. Requests
      en_us: CPU Usage vs Requests  
    description:
      zh_cn: CPU usage percentage relative to requests for the deployment
      en_us: CPU usage percentage relative to requests for the deployment
    generator: |
      sum by (namespace, deployment) (
        rate(container_cpu_usage_seconds_total{pod!~"POD"}[5m])
      ) / sum by (namespace, deployment) (
        kube_pod_container_resource_requests_cpu_cores{pod!~"POD"}
      ) * 100
    data_format: percent
    unit: ''
    golden_metric: true
    interval_us: [15000000]
    type: gauge

• Request count metric.

  - name: request_count
    display_name: Request Count
    description: Request count refers to the total number of calls to a specific application or API.
    generator: 'sum_over_time_lcro(arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc{callKind=~"http|rpc|custom_entry|server|consumer|schedule"}[60s])'
    aggregator: sum
    data_format: KMB
    golden_metric: true
    interval_us: 15000000

Variable substitution mechanism

• In the UModel system, MetricSets that use Prometheus syntax have built-in label filter injection. After you filter label values in `labels`, the system automatically injects the filter condition expression into the `generator`.

• The `generator` in a metric supports join operations on multiple metrics. However, not all metrics support every label injection. For example, some high-level Kubernetes metrics require multi-metric joins for data enrichment. Use the variable substitution mechanism to handle differences in label fields.

# Example: The node label exists only in kube_pod_info
kube_pod_info{node=~"${{node|.*}}"}

Label association example

# Metrics related to memory resources, associating pods and deployments through a ReplicaSet
- aggregator: sum
  data_format: byte
  description:
      en_us: Total memory requests for all pods in the deployment
      zh_cn: Total memory requests for all pods in the deployment
  display_name:
      en_us: Deployment Memory Requests Total
      zh_cn: Deployment Memory Requests Total
  generator: sum by (namespace, deployment) (kube_pod_container_resource_requests_memory_bytes{pod!~"POD"} * on (pod, namespace) group_left (deployment) (max by (pod, namespace, deployment) (label_replace(kube_pod_info{created_by_kind="ReplicaSet"}, "replicaset", "$1", "created_by_name", "(.*)") * on (namespace, replicaset) group_left (deployment) label_replace(kube_replicaset_owner{owner_kind="Deployment", owner_name=~"${{deployment|.*}}"}, "deployment", "$1", "owner_name", "(.*)"))))
  golden_metric: false
  interval_us:
  - 15000000
  launch_stage: ga
  name: deployment_memory_requests_total
  type: gauge
  unit: ''

Design principles

1. Clear semantics: The metric name should clearly express its business meaning.

2. Reasonable labels: Avoid high-cardinality labels to maintain query performance.

3. Complete calculation: The `generator` must contain the complete business logic.

4. Standard format: Strictly use the `data_format` enumeration values that are defined in the Schema.

5. Correct aggregation: Select the appropriate `aggregator` based on the metric's attributes.

Performance optimization

1. Label filtering: Apply label filters in the early stages of a query.

2. Aggregation optimization: Use functions such as sum by to reduce the number of time series.

3. Dynamic labels: Use `dynamic: true` to improve label management efficiency.