DocsICP FilingConsole
Documentation
Home Page Container Service for Kubernetes ACK Managed and Dedicated Clusters User Guide Auto scaling Auto scaling FAQs Workload scaling FAQ

Workload scaling FAQ

Updated at:
Copy as MD
Product
My Favorites

Resolve common HPA and CronHPA issues including metric retrieval failures, over-scaling, and threshold behavior.

Contents

Why does the current field in HPA metrics show as unknown?

When the HPA current field shows unknown, the kube-controller-manager cannot access the metric data source, causing HPA scaling to fail.

Name:                                                  kubernetes-tutorial-deployment
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Mon, 10 Jun 2019 11:46:48  0530
Reference:                                             Deployment/kubernetes-tutorial-deployment
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 2%
Min replicas:                                          1
Max replicas:                                          4
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                      From                       Message
  ----     ------                   ----                     ----                       -------
  Warning  FailedGetResourceMetric  3m3s (x1009 over 4h18m)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Cause 1: The resource metrics data source is unavailable

Run kubectl top pod to check if it returns data. If no pods return data, run kubectl get apiservice to check the Resource Metrics data source status. Example output:

Sample output

NAME                                   SERVICE                      AVAILABLE   AGE
v1.                                    Local                        True        29h
v1.admissionregistration.k8s.io        Local                        True        29h
v1.apiextensions.k8s.io                Local                        True        29h
v1.apps                                Local                        True        29h
v1.authentication.k8s.io               Local                        True        29h
v1.authorization.k8s.io                Local                        True        29h
v1.autoscaling                         Local                        True        29h
v1.batch                               Local                        True        29h
v1.coordination.k8s.io                 Local                        True        29h
v1.monitoring.coreos.com               Local                        True        29h
v1.networking.k8s.io                   Local                        True        29h
v1.rbac.authorization.k8s.io           Local                        True        29h
v1.scheduling.k8s.io                   Local                        True        29h
v1.storage.k8s.io                      Local                        True        29h
v1alpha1.argoproj.io                   Local                        True        29h
v1alpha1.fedlearner.k8s.io             Local                        True        5h11m
v1beta1.admissionregistration.k8s.io   Local                        True        29h
v1beta1.alicloud.com                   Local                        True        29h
v1beta1.apiextensions.k8s.io           Local                        True        29h
v1beta1.apps                           Local                        True        29h
v1beta1.authentication.k8s.io          Local                        True        29h
v1beta1.authorization.k8s.io           Local                        True        29h
v1beta1.batch                          Local                        True        29h
v1beta1.certificates.k8s.io            Local                        True        29h
v1beta1.coordination.k8s.io            Local                        True        29h
v1beta1.events.k8s.io                  Local                        True        29h
v1beta1.extensions                     Local                        True        29h
...
[v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        29h]
...
v1beta1.networking.k8s.io              Local                        True        29h
v1beta1.node.k8s.io                    Local                        True        29h
v1beta1.policy                         Local                        True        29h
v1beta1.rbac.authorization.k8s.io      Local                        True        29h
v1beta1.scheduling.k8s.io              Local                        True        29h
v1beta1.storage.k8s.io                 Local                        True        29h
v1beta2.apps                           Local                        True        29h
v2beta1.autoscaling                    Local                        True        29h
v2beta2.autoscaling                    Local                        True        29h

If the API Service for v1beta1.metrics.k8s.io is not kube-system/metrics-server, it may have been overwritten by Prometheus Operator. Restore it by deploying this YAML:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

If this is not the issue, go to Operations > Add-ons for your cluster and verify that metrics-server is installed.

Cause 2: Metrics cannot be retrieved during a rolling update or scale-out

After a pod is created or updated, metrics-server needs time to collect metrics. Wait about two minutes before checking again.

Cause 3: The request field is not configured

By default, HPA uses actual utilization/request as the utilization value. Check whether the pod's resource field contains the request field.

Cause 4: The metric name is incorrect

Check that the metric name and case are correct. For example, writing cpu as CPU causes the current field to show unknown.

Troubleshooting HPA scaling failures

When metric retrieval fails, the HPA current field shows unknown and scaling stops. See Node Autoscaling FAQ to troubleshoot.

Why does HPA create extra pods during a rolling update?

During a rolling update, the controller-manager reports zero values for pods without metrics, which can cause HPA to create excess pods (over-scaling). Use the following configurations to prevent this.

Cluster-level configuration

Upgrade the ACK metrics-server to the latest version and enable this startup parameter.

This affects all workloads in the cluster.

# Add the following option to the metrics-server startup parameters.
--enable-hpa-rolling-update-skipped=true

Workload-level configuration

To prevent over-scaling for a specific workload, use one of the following methods.

  • Pause HPA evaluation during rolling updates by adding this annotation to the pod template:

    # Add this annotation to spec.template.metadata.annotations of the workload to temporarily pause HPA evaluation during a rolling update.
HPARollingUpdateSkipped: "true"

    Sample code

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-basic
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template: 
    metadata:
      labels:
        app: nginx
      annotations:
        HPARollingUpdateSkipped: "true"  # Skips the HPA evaluation during a rolling update.
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

  • Set a warm-up period after application start by adding this annotation to the pod template:

    # Add this annotation to spec.template.metadata.annotations of the workload to skip a set warm-up period.
HPAScaleUpDelay: 3m # 3m is an example. Set the period based on your requirements.

    Sample code

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-basic
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template: 
    metadata:
      labels:
        app: nginx
      annotations:
        HPAScaleUpDelay: 3m  # 'm' stands for minutes. This setting makes the HPA effective 3 minutes after the pod is created. Supported units are s (seconds) and m (minutes).
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

Why does HPA not scale when a threshold is reached?

HPA does not scale solely based on whether CPU or memory crosses a threshold. It also checks whether a scaling action would be immediately reversed, preventing thrashing.

For example, with an 80% scale-out threshold and two pods at 70% CPU each, HPA will not scale in because reducing to one pod would push CPU above 80% and immediately trigger scale-out.

How to configure the HPA metric collection interval

For metric-server versions greater than v0.2.1-b46d98c-aliyun, set the --metric-resolution startup parameter. For example: --metric-resolution=15s.

Is CronHPA compatible with HPA? How does it work?

Yes. In ACK, CronHPA sets its scaleTargetRef to the HPA and scales through it rather than directly adjusting a Deployment's replica count, preventing conflicts between the two autoscalers. See Enable collaboration between CronHPA and HPA.

How to prevent HPA over-scaling from initial resource spikes

Applications in languages like Java can spike CPU and memory during warm-up after container start, triggering unnecessary HPA scale-outs. To prevent this, upgrade the ACK metrics-server to 0.3.9.6 or later and add an annotation to your pod specification. See Upgrade the metrics-server component before you upgrade your cluster to v1.12.

Sample Deployment with the annotation:

Sample YAML

## Example of a Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-basic
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        HPAScaleUpDelay: 3m # 'm' stands for minutes. This setting makes the HPA effective 3 minutes after the pod is created. Supported units are s (seconds) and m (minutes).
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9 # Replace it with your exact <image_name:tags>.
        ports:
        - containerPort: 80

Why did HPA scale when the metric value was below the threshold in the audit logs?

Cause

The Horizontal Pod Autoscaler calculates the desired number of replicas based on the ratio between the current and desired metrics by using the following formula: desired number of replicas = ceil(current number of replicas × (current metric / desired metric)).

The desired replica count depends on the current replica count, current metric, and desired metric. For resource metrics, HPA obtains the scale subresource (subResources) of the scaleTargetRef object and converts the Selector from the status of the scale object into a labelselector to match pods. If the matched pods do not all belong to the scaleTargetRef object, the calculated replica count may be incorrect (for example, scaling up when the metric is below the threshold).

Common reasons for an inaccurate pod count include:

  • A rolling update is in progress.

  • Pods outside the scaleTargetRef object share the same label. Check for them:

    kubectl get pods -n {namespace_name} -l {value_of_scale_subresource_status.selector}

Solution

Can HPA control the scale-in order of pods?

No. HPA only adjusts the replica count, not which pods to terminate. The managing controller (such as a Deployment) determines termination order and graceful shutdown.

However, in mixed environments with node types such as ECS, ACS, and ECI, or multiple node pools, use a ResourcePolicy to control scale-in priority — for example, scale in ECI pods before ECS pods. See Customize elastic resource priority scheduling.

What do HPA utilization metric units mean?

Usage metrics are unitless integers or integers in m units (1000m = 1). For example, 70000m equals 70.

What should I do if the target column shows unknown after I run kubectl get hpa?

To troubleshoot:

  1. Run kubectl describe hpa <hpa_name> to check why the HPA failed.

    • If the Conditions field indicates that AbleToScale is False, verify that the Deployment is running correctly.

    • If the Conditions field indicates that ScalingActive is False, proceed to the next step.

  2. Run kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/". If Error from server (NotFound): the server could not find the requested resource is returned, check alibaba-cloud-metrics-adapter status.

    If alibaba-cloud-metrics-adapter is running correctly, check if the HPA metric is Ingress-related. If so, deploy the SLS component first. See Collect and analyze Nginx Ingress access logs.

  3. Verify that the HPA metrics are entered correctly. The value of sls.ingress.route is in the format <namespace>-<svc>-<port>.

    • namespace: The namespace of the Ingress.

    • svc: The Ingress Service name.

    • port: The Ingress Service port name.

How to find metrics supported by HPA

See Alibaba Cloud HPA metrics for the full list. Common metrics:

Metric

Description

Additional parameter

sls_ingress_qps

QPS for the specified Ingress route.

sls.ingress.route

sls_alb_ingress_qps

QPS for the specified ALB Ingress route.

sls.ingress.route

sls_ingress_latency_avg

Average latency for all requests.

sls.ingress.route

sls_ingress_latency_p50

50th percentile latency of requests.

sls.ingress.route

sls_ingress_latency_p95

95th percentile latency of requests.

sls.ingress.route

sls_ingress_latency_p99

99th percentile latency of requests.

sls.ingress.route

sls_ingress_latency_p9999

99.99th percentile latency of requests.

sls.ingress.route

sls_ingress_inflow

Inbound bandwidth of the Ingress.

sls.ingress.route

Autoscaling with custom Nginx Ingress log formats

To use SLS Ingress metrics for horizontal pod scaling, see Scale pods based on Nginx Ingress metrics. Nginx Ingress log collection to SLS must be enabled in your cluster.

  • SLS is enabled by default during cluster creation. With default settings, access log dashboards and Nginx Ingress monitoring are available in the SLS console.

  • If you disabled SLS during cluster creation, re-enable and configure it. See Collect and analyze Nginx Ingress access logs.

  • If you customized the Nginx Ingress log format, update the processor_regex section in the CRD configuration. The default AliyunLogConfig CRD only matches the default Ingress Controller log format. See Collect container logs by using a DaemonSet-CRD.

How to retrieve the QPS metric sls_ingress_qps using the CLI?

Retrieve the sls_ingress_qps metric:

kubectl get --raw  "/apis/external.metrics.k8s.io/v1beta1/namespaces/*/sls_ingress_qps?labelSelector=sls.project={{SLS_Project}},sls.logstore=nginx-ingress"

{{SLS_Project}} is the SLS Project name for the ACK cluster. Default: k8s-log-{{ClusterId}}, where {{ClusterId}} is the cluster ID.

If this error is returned:

Error from server: {
    "httpCode": 400,
    "errorCode": "ParameterInvalid",
    "errorMessage": "key (slb_pool_name) is not config as key value config,if symbol : is  in your log,please wrap : with quotation mark \"",
    "requestID": "xxxxxxx"
}

This means no data is available for the metric, which occurs when querying an ALB Ingress metric (such as sls_alb_ingress_qps) without a configured ALB Ingress.

If a result like this is returned:

{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "sls_ingress_qps",
      "timestamp": "2025-02-26T16:45:00Z", 
      "value": "50",   # The QPS value
      "metricLabels": {
        "sls.project": "your-sls-project-name",
        "sls.logstore": "nginx-ingress"
      }
    }
  ]
}

This confirms the metric was retrieved. value is the QPS value.

Failed to pull alibaba-cloud-metrics-adapter image

Symptom

When you upgrade the ack-alibaba-cloud-metrics-adapter component to version 1.3.7, the image pull fails with the following error:

Failed to pull image "registry-<region-id>-vpc.ack.aliyuncs.com/acs/alibaba-cloud-metrics-adapter-amd64:v0.2.9-ba634de-aliyun".

Cause

ack-alibaba-cloud-metrics-adapter does not currently support in-place upgrades.

Solution

To upgrade:

  1. Back up the current component configuration.

  2. Uninstall the old version of the component.

  3. Install the latest version with the backed-up configuration.

Important

During this process, scaling for related HPA objects is paused because metric collection stops.

Previous：Node instant scaling FAQNext：GPU