kube-apiserver: Metrics and dashboard

Name

PromQL

Description

API QPS

sum(irate(apiserver_request_total[$interval]))

Total QPS for the API Server.

Read request success rate

sum(irate(apiserver_request_total{code=~"20.*",verb=~"GET|LIST"}[$interval]))/sum(irate(apiserver_request_total{verb=~"GET|LIST"}[$interval]))

Read request success rate for the API Server.

Write request success rate

sum(irate(apiserver_request_total{code=~"20.*",verb!~"GET|LIST|WATCH|CONNECT"}[$interval]))/sum(irate(apiserver_request_total{verb!~"GET|LIST|WATCH|CONNECT"}[$interval]))

Write request success rate for the API Server.

Number of in-flight read requests

sum(apiserver_current_inflight_requests{requestKind="readOnly"})

In-flight read requests the API Server is currently processing.

Number of in-flight write requests

sum(apiserver_current_inflight_requests{requestKind="mutating"})

In-flight write requests the API Server is currently processing.

Request throttling rate

sum(irate(apiserver_dropped_requests_total[$interval]))

The rate at which the API Server's throttling policy drops requests.

Percentage of total requests dropped by the API server throttling policy.

Name

PromQL

Description

GET read request latency

histogram_quantile($quantile, sum(irate(apiserver_request_duration_seconds_bucket{verb="GET",resource!="",subresource!~"log|proxy"}[$interval])) by (pod, verb, resource, subresource, scope, le))

GET request response time, broken down by API Server pod, resource, and scope.

LIST read request latency

histogram_quantile($quantile, sum(irate(apiserver_request_duration_seconds_bucket{verb="LIST"}[$interval])) by (pod_name, verb, resource, scope, le))

LIST request response time, broken down by API Server pod, resource, and scope.

Write request latency

histogram_quantile($quantile, sum(irate(apiserver_request_duration_seconds_bucket{verb!~"GET|WATCH|LIST|CONNECT"}[$interval])) by (cluster, pod_name, verb, resource, scope, le))

Mutating request response time, broken down by API Server pod, verb (excluding GET, WATCH, LIST, CONNECT), resource, and scope.

Number of in-flight read requests

apiserver_current_inflight_requests{request_kind="readOnly"}

Read requests the API Server is currently processing.

Number of in-flight write requests

apiserver_current_inflight_requests{request_kind="mutating"}

Write requests the API Server is currently processing.

Request throttling rate

sum(irate(apiserver_dropped_requests_total{request_kind="readOnly"}[$interval])) by (name)

sum(irate(apiserver_dropped_requests_total{request_kind="mutating"}[$interval])) by (name)

Request throttling rate for the API Server. No data or 0 indicates no throttling.

Name

PromQL

Description

Memory usage

memory_utilization_byte{container="kube-apiserver"}

Memory usage of the API Server. Unit: bytes.

CPU usage

cpu_utilization_core{container="kube-apiserver"}*1000

CPU usage of the API Server. Unit: millicores.

Resource object count

• max by(resource)(apiserver_storage_objects)

• max by(resource)(etcd_object_counts)

• For ACK clusters running v1.22 or later, the metric name is apiserver_storage_objects.

• For ACK clusters running a version earlier than 1.22, the metric name is etcd_object_counts.

Memory utilization level

• resource_utilization_level{resource="memory",container="kube-apiserver",utilization_level="high"}

• resource_utilization_level{resource="memory",container="kube-apiserver",utilization_level="normal"}

• If resource_utilization_level{utilization_level="high",...} is 1, the container resource utilization level is ≥ 80%.

• If resource_utilization_level{utilization_level="normal",...} is 1, the container resource utilization level is < 80%.

CPU utilization level

• resource_utilization_level{resource="cpu",container="kube-apiserver",utilization_level="high"}

• resource_utilization_level{resource="cpu",container="kube-apiserver",utilization_level="normal"}

Name

PromQL

Description

QPS analysis by verb

sum(irate(apiserver_request_total{verb=~"$verb"}[$interval]))by(verb)

The request QPS, broken down by verb.

QPS analysis by verb and resource

sum(irate(apiserver_request_total{verb=~"$verb",resource=~"$resource"}[$interval]))by(verb,resource)

The request QPS, broken down by verb and resource.

Request latency analysis by verb

histogram_quantile($quantile, sum(irate(apiserver_request_duration_seconds_bucket{verb=~"$verb", verb!~"WATCH|CONNECT",resource!=""}[$interval])) by (le,verb))

The request latency, broken down by verb.

Request latency analysis by verb and resource

histogram_quantile($quantile, sum(irate(apiserver_request_duration_seconds_bucket{verb=~"$verb", verb!~"WATCH|CONNECT", resource=~"$resource",resource!=""}[$interval])) by (le,verb,resource))

The request latency, broken down by verb and resource.

QPS of read requests with non-2xx responses

sum(irate(apiserver_request_total{verb=~"GET|LIST",resource=~"$resource",code!~"2.*"}[$interval])) by (verb,resource,code)

The QPS of read requests that returned a non-2xx status code, such as 4xx or 5xx.

QPS of write requests with non-2xx responses

sum(irate(apiserver_request_total{verb!~"GET|LIST|WATCH",verb=~"$verb",resource=~"$resource",code!~"2.*"}[$interval])) by (verb,resource,code)

The QPS of write requests that returned a non-2xx status code, such as 4xx or 5xx.

API Server to etcd request latency

histogram_quantile($quantile, sum(irate(etcd_request_duration_seconds_bucket[$interval])) by (le,operation,type,instance))

The latency of requests from the API Server to etcd.

Name

PromQL

Description

APF request concurrency limit (by PL)

sum by(priority_level) (apiserver_flowcontrol_request_concurrency_limit)

APF request concurrency limit, broken down by PL or Instance + PL. The theoretical maximum concurrent requests per priority queue.

apiserver_flowcontrol_request_concurrency_limit is deprecated in Kubernetes 1.30 and removed in 1.31. For clusters running v1.31 or later, use apiserver_flowcontrol_nominal_limit_seats instead.

APF request concurrency limit (by Instance + PL)

sum by(instance,priority_level) (apiserver_flowcontrol_request_concurrency_limit)

Number of current APF executing requests (by FS + PL)

sum by(flow_schema,priority_level) (apiserver_flowcontrol_current_executing_requests)

Requests currently executing under APF, broken down by FS + PL or Instance + FS + PL.

Number of current APF executing requests (by Instance + FS + PL)

sum by(instance,flow_schema,priority_level)(apiserver_flowcontrol_current_executing_requests)

Number of current APF in-queue requests (by FS + PL)

sum by(flow_schema,priority_level) (apiserver_flowcontrol_current_inqueue_requests)

Requests waiting in queue, broken down by FS + PL or Instance + FS + PL.

Number of current APF in-queue requests (by Instance + FS + PL)

sum by(instance,flow_schema,priority_level) (apiserver_flowcontrol_current_inqueue_requests)

APF nominal limit seats

sum by(instance,priority_level) (apiserver_flowcontrol_nominal_limit_seats)

APF seat metrics by Instance + PL:

• Nominal limit seats: nominal maximum concurrency in seats per priority queue.

• Current limit seats: maximum concurrent seats after dynamic adjustments per priority queue.

• Executing seats: seats consumed by currently executing requests per priority queue.

• In-queue seats: seats consumed by waiting requests per priority queue.

APF current limit seats

sum by(instance,priority_level) (apiserver_flowcontrol_current_limit_seats)

APF current executing seats

sum by(instance,priority_level) (apiserver_flowcontrol_current_executing_seats)

APF current in-queue seats

sum by(instance,priority_level) (apiserver_flowcontrol_current_inqueue_seats)

APF request execution time

histogram_quantile($quantile, sum(irate(apiserver_flowcontrol_request_execution_seconds_bucket[$interval])) by (le,instance, flow_schema,priority_level))

Time from request execution start to completion.

APF request wait time

histogram_quantile($quantile, sum(irate(apiserver_flowcontrol_request_wait_seconds_bucket[$interval])) by (le,instance, flow_schema,priority_level))

Time a request waits in queue before execution.

QPS of successfully dispatched APF requests

sum(irate(apiserver_flowcontrol_dispatched_requests_total[$interval]))by(instance,flow_schema,priority_level)

QPS of successfully dispatched requests.

QPS of rejected APF requests

sum(irate(apiserver_flowcontrol_rejected_requests_total[$interval]))by(instance,flow_schema,priority_level)

QPS of requests rejected for exceeding concurrency limit or queue capacity.

Name

PromQL

Description

Admission controller latency [admit]

histogram_quantile($quantile, sum by(operation, name, le, type, rejected) (irate(apiserver_admission_controller_admission_duration_seconds_bucket{type="admit"}[$interval])) )

The name, operation, rejection status, and execution time of admit type admission controllers.

Histogram buckets:{0.005, 0.025, 0.1, 0.5, 2.5}. Unit: seconds.

Admission controller latency [validate]

histogram_quantile($quantile, sum by(operation, name, le, type, rejected) (irate(apiserver_admission_controller_admission_duration_seconds_bucket{type="validate"}[$interval])) )

The name, operation, rejection status, and execution time of validate type admission controllers.

Histogram buckets:{0.005, 0.025, 0.1, 0.5, 2.5}. Unit: seconds.

Admission webhook latency [admit]

histogram_quantile($quantile, sum by(operation, name, le, type, rejected) (irate(apiserver_admission_webhook_admission_duration_seconds_bucket{type="admit"}[$interval])) )

The name, operation, rejection status, and execution time of admit type webhooks.

Histogram buckets:{0.005, 0.025, 0.1, 0.5, 2.5}. Unit: seconds.

Admission webhook latency [validating]

histogram_quantile($quantile, sum by(operation, name, le, type, rejected) (irate(apiserver_admission_webhook_admission_duration_seconds_bucket{type="validating"}[$interval])) )

The name, operation, rejection status, and execution time of validating type webhooks.

Histogram buckets:{0.005, 0.025, 0.1, 0.5, 2.5}. Unit: seconds.

Admission webhook request QPS

sum(irate(apiserver_admission_webhook_admission_duration_seconds_count[$interval]))by(name,operation,type,rejected)

The request QPS of admission webhooks.

Name

PromQL

Description

QPS analysis by client

sum(irate(apiserver_request_total{client!=""}[$interval])) by (client)

QPS by client, showing which clients access the API Server.

QPS analysis by verb, resource, and client

sum(irate(apiserver_request_total{client!="",verb=~"$verb", resource=~"$resource"}[$interval]))by(verb,resource,client)

The QPS of API Server requests, broken down by verb, resource, and client.

QPS of LIST requests without resourceVersion

sum(irate(apiserver_request_no_resourceversion_list_total[$interval]))by(resource,client)

• The QPS of LIST requests that do not include the resourceVersion field, broken down by resource and client.

• Analyze LIST requests from the API Server to etcd to optimize API Server client behavior.

Normal

Abnormal

Description

The read request success rate and write request success rate should be close to 100%.

The read request success rate and write request success rate are consistently below 90%.

Many requests return non-2xx status codes.

Normal

Abnormal

Description

The GET read request latency P[0.9], LIST read request latency P[0.9], and write request latency P[0.9] correlate with cluster size and number of resources accessed. No fixed threshold — latency is acceptable if it does not affect your applications. Typically, GET read request latency P[0.9] and write request latency P[0.9] under 1s, and LIST read request latency P[0.9] under 5s are normal.

• The GET read request latency P[0.9] or write request latency P[0.9] exceeds 1s.

• The LIST read request latency P[0.9] exceeds 5s.

If request latency is high, you should rule out factors such as a large number of cluster resources or slow webhook calls.

Normal

Abnormal

Description

Typically, number of read requests processed and number of write requests processed are under 100, and request limit rate is 0.

• The number of read requests processed or number of write requests processed exceeds 100.

• The request limit rate is greater than 0.

If the request queue has a backlog, rule out factors such as a sudden request surge or slow webhooks. If queue capacity is exceeded, the API server throttles requests, causing the request limit rate to rise above 0 and affecting cluster stability.

Normal

Abnormal

Description

The admission webhook latency should be less than 0.5s.

The admission webhook latency persistently exceeds 0.5s.

Slow webhook responses affect API server response latency.