HPA FAQs
Use this topic to diagnose and resolve common Horizontal Pod Autoscaler (HPA) issues in ACK, including metric collection failures, unexpected scaling behavior, and multi-metric configuration.
Why does HPA fail to collect resource metrics?
The most common sign is a FailedGetResourceMetric warning in the Conditions section of kubectl describe hpa output:
Name: kubernetes-tutorial-deployment
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Mon, 10 Jun 2019 11:46:48 +0530
Reference: Deployment/kubernetes-tutorial-deployment
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 2%
Min replicas: 1
Max replicas: 4
Deployment pods: 1 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 3m3s (x1009 over 4h18m) horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)Understanding the Conditions fields helps narrow down the problem quickly:
| Field | Meaning | When False |
|---|---|---|
AbleToScale | HPA can fetch and update the scale target | Typically indicates a backoff or permission issue |
ScalingActive | HPA can successfully calculate a replica count | Metric collection is failing — this is what FailedGetResourceMetric reports |
ScalingLimited | The desired replica count was capped by the configured minimum or maximum | The current replica count is at a boundary |
When ScalingActive is False with FailedGetResourceMetric, work through the following checks.
Check 1: Verify metrics-server is serving data
Run kubectl top pod. If no data is returned, metrics-server is not serving metrics. Then run kubectl get apiservice and look for v1beta1.metrics.k8s.io in the output:
NAME SERVICE AVAILABLE AGE
v1. Local True 29h
v1.admissionregistration.k8s.io Local True 29h
v1.apiextensions.k8s.io Local True 29h
v1.apps Local True 29h
v1.authentication.k8s.io Local True 29h
v1.authorization.k8s.io Local True 29h
v1.autoscaling Local True 29h
v1.batch Local True 29h
v1.coordination.k8s.io Local True 29h
v1.monitoring.coreos.com Local True 29h
v1.networking.k8s.io Local True 29h
v1.rbac.authorization.k8s.io Local True 29h
v1.scheduling.k8s.io Local True 29h
v1.storage.k8s.io Local True 29h
v1alpha1.argoproj.io Local True 29h
v1alpha1.fedlearner.k8s.io Local True 5h11m
v1beta1.admissionregistration.k8s.io Local True 29h
v1beta1.alicloud.com Local True 29h
v1beta1.apiextensions.k8s.io Local True 29h
v1beta1.apps Local True 29h
v1beta1.authentication.k8s.io Local True 29h
v1beta1.authorization.k8s.io Local True 29h
v1beta1.batch Local True 29h
v1beta1.certificates.k8s.io Local True 29h
v1beta1.coordination.k8s.io Local True 29h
v1beta1.events.k8s.io Local True 29h
v1beta1.extensions Local True 29h
...
[v1beta1.metrics.k8s.io kube-system/metrics-server True 29h]
...
v1beta1.networking.k8s.io Local True 29h
v1beta1.node.k8s.io Local True 29h
v1beta1.policy Local True 29h
v1beta1.rbac.authorization.k8s.io Local True 29h
v1beta1.scheduling.k8s.io Local True 29h
v1beta1.storage.k8s.io Local True 29h
v1beta2.apps Local True 29h
v2beta1.autoscaling Local True 29h
v2beta2.autoscaling Local True 29hThe SERVICE for v1beta1.metrics.k8s.io must be kube-system/metrics-server. If it points elsewhere, Prometheus Operator may have overwritten the APIService registration. Redeploy metrics-server using this YAML:
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
name: v1beta1.metrics.k8s.io
spec:
service:
name: metrics-server
namespace: kube-system
group: metrics.k8s.io
version: v1beta1
insecureSkipTLSVerify: true
groupPriorityMinimum: 100
versionPriority: 100Check 2: Account for the metrics-server startup delay
metrics-server collects metrics at 1-second intervals, but it needs a few seconds to start collecting after a rolling update or scale out. If you just performed either, wait 2 seconds before checking metrics again.
Check 3: Confirm resource requests are set on the pod
HPA calculates CPU and memory usage as used resource / requested resource. If the requests field is missing from the pod spec, HPA cannot compute a ratio and will fail silently. Make sure resources.requests is defined for every container in the pod.
Why does HPA add too many pods during a rolling update?
During a rolling update, kube-controller-manager treats pods whose metrics cannot be collected as having zero usage. This zero-filling inflates the calculated replica count, causing HPA to scale out more pods than needed.
Update metrics-server to the latest version and add the following flag to its startup configuration:
--enable-hpa-rolling-update-skipped=trueWhy doesn't HPA scale even when the threshold is reached?
HPA scaling conditions are not strictly based on exceeding or falling below thresholds. HPA also takes other factors into consideration when it scales pods. For example, it checks whether the current scale-out event would trigger a scale-in activity, or whether the scale-in event would trigger a scale-out activity. This avoids repetitive scaling and prevents unnecessary resource consumption.
If HPA is not scaling as expected, check whether the current scaling direction would immediately trigger the reverse action — HPA checks this to prevent oscillation.
How do I set the data collection interval for HPA?
For metrics-server versions later than 0.2.1-b46d98c-aliyun, add the --metric-resolution flag to the metrics-server startup configuration:
--metric-resolution=15sCan the memory HPA trigger a scale-in after the CPU HPA triggers a scale-out?
Yes. If you configure separate HPA objects for CPU and memory, they operate independently. A CPU HPA scale-out does not prevent the memory HPA from triggering a scale-in.
To avoid this conflict, use a single HPA that monitors both CPU and memory. When one HPA watches multiple metrics, it evaluates each metric independently and applies the maximum recommended replica count across all metrics:
For scale-out: HPA scales to whichever metric recommends the most pods.
For scale-in: HPA retains whichever metric recommends keeping the most pods.
This single rule — take the maximum — covers both directions and keeps workloads stable during metric fluctuations.