Troubleshoot pod issues

Error message

Description

Solution

no nodes available to schedule pods.

The cluster has no available nodes for pod scheduling.

1. Check if any nodes in the cluster are in the NotReady state. If a node is NotReady, inspect and repair it.

2. Check if the pod defines a nodeSelector, nodeAffinity, or taint tolerations. If no such scheduling constraints are defined, consider adding more nodes to the node pool.

• 0/x nodes are available: x Insufficient cpu.

• 0/x nodes are available: x Insufficient memory.

No available nodes in the cluster can meet the pod's CPU or memory resource requests.

A node is unschedulable when its total allocated requests reach capacity, even if actual utilization is low.

On the target cluster's details page, go to Nodes > Nodes and check the CPU or memory requests allocation rate for the target node. You can hover over the allocation rate to view the specific resource allocation values.

To view detailed node resource usage, see Use kubectl to view node resource usage.

• Optimize resource configuration:

  • If a node's resource usage is consistently lower than its requests, resources are being wasted. Lower the workload's requests configuration. See Set CPU and memory resource limits for a container.

    You can enable resource profiling to obtain the recommended requests configuration.

  • Enable Horizontal Pod Autoscaler (HPA) to scale down replicas during off-peak hours.

• Clean up unnecessary workloads: Decommission or scale down non-essential pods.

• Scale out the node pool: If the resource usage on the target nodes is consistently high, the nodes are saturated. You can scale out the node pool.

x node(s) didn't match pod's node affinity/selector.

The existing nodes do not match the pod's node affinity policy (nodeAffinity/nodeSelector). See Assigning Pods to Nodes.

1. View all labels on a node.

   Console

   1. On the target cluster's details page, go to Nodes > Nodes.

   2. On the Nodes page, find the target node, and in the Actions column, click More > Manage Labels and Taints to view its labels.

   Kubectl

   Replace <YOUR_NODE_NAME> with your actual node name.

   kubectl get node <YOUR_NODE_NAME> --show-labels

2. Check and adjust the node affinity rule for the workload (deployment).

   Console

   When creating a new workload:

   1. On the Advanced page for creating a Create Deployment, find Node Affinity in the Scheduling section, and click Add.

   2. Configure either Required (hard affinity) or Optional (soft affinity) based on your business needs. Multiple Selector have a logical AND relationship, while multiple Rule have a logical OR relationship.

   For existing workloads:

   1. On the Nodes > Nodes page, click > Node Affinity in the Actions column of the target Deployment.

   2. The configuration method is the same as described above.

   YAML example

   NodeAffinity

   Affinity policies include hard affinity (requiredDuringSchedulingIgnoredDuringExecution), which must be met, and soft affinity (preferredDuringSchedulingIgnoredDuringExecution), which expresses a preference. The following example uses hard affinity.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: app-demo-node-affinity-deploy
     labels:
       app: demo-node-affinity
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: demo-node-affinity
     template:
       metadata:
         labels:
           app: demo-node-affinity
       spec:
         containers:
         - name: nginx
           image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
         affinity:
           nodeAffinity:
             # Hard affinity: The rule must be met.
             requiredDuringSchedulingIgnoredDuringExecution:
               nodeSelectorTerms:
               - matchExpressions:
                 - key: disktype
                   operator: In
                   values:
                   - ssd
                   - nvme  # Logic: The node's 'disktype' label must be either 'ssd' or 'nvme'.

   NodeSelector

   This provides a simple exact match. The pod is scheduled only if the node's labels meet the conditions.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: app-demo-node-selector-deploy
     labels:
       app: demo-node-selector
   spec:
     replicas: 2  
     selector:
       matchLabels:
         app: demo-node-selector  
     template:
       metadata:
         labels:
           app: demo-node-selector
       spec:
         containers:
         - name: nginx
           image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
         # The pod is scheduled only if the node has the label disktype=ssd.
         nodeSelector:
           disktype: ssd

• x node(s) didn't match pod affinity rules.

• x node(s) didn't match pod anti-affinity rules.

• Affinity rule mismatch. The pod has a pod affinity rule (for example, requiring a specific label), but no nodes host a pod with a matching label, preventing scheduling.

• Anti-affinity conflict. The pod has a pod anti-affinity rule (for example, it cannot coexist with another application), but all available nodes already host a conflicting pod, preventing scheduling.

1. View the pod labels on a node.

   Console

   1. On the target cluster's details page, go to Nodes > Nodes.

   2. On the Nodes page, click the name of the target node to view its details page. Scroll down to the Pods section to view the label values for different pods in the Label column.

   Kubectl

   • View pods and their labels on a specific node: Replace <YOUR_NAMESPACE> with your namespace name and <YOUR_NODE_NAME> with your actual node name.

     kubectl get pods -n <YOUR_NAMESPACE> --field-selector spec.nodeName=<YOUR_NODE_NAME> -o custom-columns=NAME:.metadata.name,LABELS:.metadata.labels

   • Query pods by label: Replace <LABEL> with the actual label key-value pair, such as app=nginx.

     kubectl get pods -A -l <LABEL> -o wide

2. Check and adjust the pod affinity rule for the workload (deployment).

   Console

   1. When you create a new workload, on the Create Deployment's Advanced page, find Pod Affinity/Pod Anti-affinity in the Scheduling section, and click Add.

   2. Configure either Required (hard affinity) or Optional (soft affinity) based on your business needs. Multiple Selector have a logical AND relationship, while multiple Add Rule have a logical OR relationship.

   YAML example

   Affinity policies are classified into hard affinity (requiredDuringSchedulingIgnoredDuringExecution) and soft affinity (preferredDuringSchedulingIgnoredDuringExecution). Hard affinity rules must be met, while soft affinity rules are preferred. The following example shows a configuration for a required pod affinity.

   To configure pod anti-affinity, simply replace podAffinity with podAntiAffinity.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: app-demo-podaffinity-deploy
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: demo-podaffinity
     template:
       metadata:
         labels:
           app: demo-podaffinity
       spec:
         containers:
         - name: nginx
           image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
         affinity:
           podAffinity:
             # Hard affinity: Pod must be co-located with a pod that has the 'app: nginx' label.
             requiredDuringSchedulingIgnoredDuringExecution:
             - labelSelector:
                 matchExpressions:
                 - key: app
                   operator: In
                   values:
                   - nginx
               # Topology domain scope: host-level isolation.
               topologyKey: kubernetes.io/hostname

0/x nodes are available: x node(s) had volume node affinity conflict.

Scheduling fails due to a volume node affinity conflict. This typically occurs because a cloud disk cannot be mounted across different zones.

• For a statically provisioned PV, configure the pod's node affinity to ensure it is scheduled to a node in the same zone as the PV.

• For a dynamically provisioned PV, set the volumeBindingMode of the StorageClass to WaitForFirstConsumer. This ensures that the PV is created only after the pod has been scheduled to a node, ensuring the cloud disk is created in the same zone as the pod's node.

InvalidInstanceType.NotSupportDiskCategory

The ECS instance does not support the specified cloud disk type.

See Instance families to confirm the cloud disk types supported by your ECS instance. When mounting, update the cloud disk type to one that is supported by the ECS instance.

0/x nodes are available: x node(s) had taints that the pod didn't tolerate.

The pod cannot be scheduled to a node because it lacks a toleration for one of the node's taints.

• If the taint was added manually, remove it or configure a toleration for the pod. See Taints and Tolerations and Manage node labels and taints.

• If the taint was added by the system, resolve the underlying issue below and wait for rescheduling.

  View taints added by the system

  • node.kubernetes.io/not-ready: The node is in the NotReady state.

  • node.kubernetes.io/unreachable: The node is unreachable from the node controller. This is equivalent to the node's Ready status being Unknown.

  • node.kubernetes.io/memory-pressure: The node is under memory pressure.

  • node.kubernetes.io/disk-pressure: The node is under disk pressure.

  • node.kubernetes.io/pid-pressure: The node is under PID pressure.

  • node.kubernetes.io/network-unavailable: The node's network is unavailable.

  • node.kubernetes.io/unschedulable: The node is marked as unschedulable.

0/x nodes are available: x Insufficient ephemeral-storage.

The node has insufficient ephemeral storage.

1. Check the Pod's ephemeral storage request, which is the value of spec.containers.resources.requests.ephemeral-storage in the Pod YAML. If the value is too high and exceeds the actual available capacity of the node, the Pod will fail to be scheduled.

2. Check the total ephemeral storage capacity on each node with kubectl describe node | grep -A10 Capacity. If insufficient, expand the node's disk or add more nodes.

0/x nodes are available: pod has unbound immediate persistent volume claims.

The pod failed to bind to a persistent volume claim (PVC).

Check if the PVC or PV specified by the pod has been created. Run kubectl describe pvc <pvc-name> or kubectl describe pv <pv-name> to view PVC and PV events for further diagnosis. See Storage FAQ - CSI.

Error message

Description

Suggested solution

Failed to pull image "xxx": rpc error: code = Unknown desc = Error response from daemon: Get xxx: denied:

Access to the image repository is denied because an imagePullSecret was not specified when the pod was created.

Verify that the Secret specified in the spec.imagePullSecrets field of the workload's YAML file exists.

When using ACR, use a credential helper to pull images without a password. See Pull images from the same account.

Failed to pull image "xxxx:xxx": rpc error: code = Unknown desc = Error response from daemon: Get https://xxxxxx/xxxxx/: dial tcp: lookup xxxxxxx.xxxxx: no such host

The image repository address could not be resolved when pulling an image over HTTPS.

1. Verify that the image repository address in spec.containers.image of the pod's YAML file is correct. If it is incorrect, update it.

2. If the address is correct, verify the network connectivity from the node where the pod is running to the image repository. Log on to the node (for more information, see Choose an ECS remote connection method) and run the curl -kv https://xxxxxx/xxxxx/ command to check if the address is accessible. If an error occurs, investigate for potential network issues, such as incorrect network configuration, firewall rules, or DNS resolution problems.

Failed create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "xxxxxxxxx": Error response from daemon: mkdir xxxxx: no space left on device

The node has insufficient disk space.

Log on to the node (see Choose an ECS remote connection method) and run df -h to check disk space. If the disk is full, resize it. See Step 1: Resize a cloud disk.

Failed to pull image "xxx": rpc error: code = Unknown desc = error pulling image configuration: xxx x509: certificate signed by unknown authority

The third-party image repository uses a certificate signed by an unknown or insecure Certificate Authority (CA).

1. The third-party repository should use a certificate issued by a trusted CA.

2. If you are using a private image repository, see Create an application from a private image repository.

3. If you cannot change the certificate, you can configure the node to allow pulling and pushing images from a repository that uses an insecure certificate. We recommend using this method only in test environments, as it may affect other pods on the node.

Failed to pull image "XXX": rpc error: code = Unknown desc = context canceled

The operation was canceled, possibly because the image file is too large. Kubernetes has a default timeout for pulling images. If the pull makes no progress for a specific period, Kubernetes assumes the operation has failed or is unresponsive and cancels the task.

1. Verify that imagePullPolicy is set to IfNotPresent in the pod's YAML file.

2. Log on to the node where the pod is running (for more information, see Choose an ECS remote connection method) and run docker pull or crictl pull to check if the image can be pulled.

Failed to pull image "xxxxx": rpc error: code = Unknown desc = Error response from daemon: Get https://xxxxxxx: xxxxx/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Cannot connect to the image repository due to network issues.

1. Log on to the node where the pod is running (for more information, see Choose an ECS remote connection method) and run the curl https://xxxxxx/xxxxx/ command to check if the address is accessible. If an error occurs, investigate for potential network issues, such as incorrect network configuration, firewall rules, or DNS resolution problems.

2. Verify the node's public network policy, including configurations for SNAT entries and bound Elastic IP Addresses (EIPs).

Failed to pull image "xxxx:xxx": failed to pull and unpack image "xxxx:xxx": failed to resolve reference "xxxx:xxx": failed to do request: Head "xxxx:xxx": dial tcp xxx.xxx.xx.x:xxx: i/o timeout

Connection timed out due to network issues when pulling an image from an overseas repository.

Pulling images from overseas repositories, such as Docker Hub, may fail in ACK clusters due to unstable carrier networks. To resolve this, consider the following solutions:

• Subscribe to overseas source images using Container Registry. For instructions, see Sync images by using artifact subscription.

• Create a Global Accelerator (GA) instance to pull images directly using its global acceleration network. For details, see Accelerate cross-region ACK image pulls with GA.

Too Many Requests.

Docker Hub imposes rate limits on image pull requests.

Upload the image to Container Registry (ACR) and pull it from an ACR image repository.

The status Pulling image is consistently displayed

The kubelet's image pull rate limiting mechanism may have been triggered.

Adjust the registryPullQPS (maximum QPS for the image repository) and registryBurst (maximum number of burst image pulls) using the Customize node pool kubelet configurations feature.

Error message

Description

Solution

Stuck in the Init:N/M state

The pod has M init containers; N completed, but the remaining M-N failed to start.

1. Check the pod's events and init container issues with kubectl describe pod -n <ns> <pod name>.

2. Check the logs of the unstarted init containers with kubectl logs -n <ns> <pod name> -c <container name>.

3. Review the pod's configuration, such as the health check settings, to ensure the init containers are configured correctly.

See Debug init containers.

Stuck in the Init:Error state

An init container in the pod failed to start.

Stuck in the Init:CrashLoopBackOff state

An init container in the pod failed to start and is in a restart loop.

Error message

Description

Solution

failed to allocate for range 0: no IP addresses available in range set: xx.xxx.xx.xx-xx.xx.xx.xx

This is expected behavior due to the design of the Flannel network plugin.

Upgrade the Flannel component to v0.15.1.11-7e95fe23-aliyun or later. See Flannel.

In clusters that run a Kubernetes version earlier than 1.20, an IP address leak can occur if a pod restarts repeatedly or if pods from a CronJob complete their tasks and exit quickly.

Upgrade the cluster to Kubernetes 1.20 or later (latest recommended). See Manually upgrade a cluster.

Defects in containerd and runC cause this issue.

For an emergency fix, see Why does my pod fail to start with the error "no IP addresses available in range"?

error parse config, can't found dev by mac 00:16:3e:01:c2:e8: not found

The Terway network plugin maintains an internal database on the node to track and manage elastic network interfaces (ENIs). This error occurs when the database state is inconsistent with the actual network device configuration, causing ENI allocation to fail.

1. Network interfaces load asynchronously. The interface might still be loading during CNI configuration, which triggers an automatic CNI retry. This process does not affect the final ENI allocation. Check the pod's final status to confirm success.

2. If pod creation still fails and this error persists, the driver likely failed to load the ENI due to insufficient high-order memory. Restart the ECS instance to resolve this.

• cmdAdd: error alloc ip rpc error: code = DeadlineExceeded desc = context deadline exceeded

• cmdAdd: error alloc ip rpc error: code = Unknown desc = error wait pod eni info, timed out waiting for the condition

The Terway network plugin may have failed to request an IP address from the vSwitch.

1. View the logs of the Terway container within the Terway component pod on the node to check the ENI allocation process.

2. View ENI information for the Terway pod with kubectl logs -n kube-system  <terwayPodName > -c terway | grep <podName>. Obtain the Request ID and OpenAPI error message.

3. Use the Request ID and error message to investigate the failure.

Error message

Description

Solution

The log contains exit(0).

1. Log on to the node where the abnormal workload is deployed.

2. Use docker ps -a | grep $podName to check. If the container has no persistent process, it exits with status code 0.

The pod's events show Liveness probe failed: ....

The liveness probe failed, causing the application to restart.

• Liveness probe configuration: On the Edit page of the target Workloads, verify that the health check request path (for example, /healthz) and port match those that the application provides. Increase the Initial Delay (s) to ensure the liveness probe starts only after the application has fully launched.

  You can temporarily disable the Liveness. Then, access the pod terminal or its host node and use a command, such as curl, to verify that the health check method works correctly.

• Troubleshoot application issues: Investigate the issue by checking the pod's Events and Log. Select Show the log of the last container exit.

The pod's events show Startup probe failed: ....

The startup probe failed, causing the application to restart.

• Startup probe configuration: On the Edit page of the target Workloads, verify that the health check request path (for example, /healthz) and port match those that the application provides. If the application takes a long time to start, increase the Unhealthy Threshold to prevent premature restarts.

  You can temporarily disable the Startup. Then, access the pod terminal or its host node and use a command, such as curl, to verify that the health check method works correctly.

• Troubleshoot application issues: Investigate the issue by checking the pod's Events and Logs. Select Show the log of the last container exit.

The pod log contains no space left on device.

Insufficient cloud disk space.

• Resize the cloud disk.

• Clean up unnecessary images to free up disk space, and configure imageGCHighThresholdPercent to set the threshold for image garbage collection on the node.

Startup fails without event information.

This issue occurs when a container requires more resources than its declared limits, causing it to fail.

Check whether the pod's resource configuration is correct. You can enable resource profiling to get recommended Request and Limit configurations for the container.

The pod log shows Address already in use.

A port conflict exists between containers in the same pod.

1. Check whether the pod is configured with hostNetwork: true. This setting causes containers in the pod to share the host's network namespace and port space. If this is not required, change it to hostNetwork: false.

2. If the pod requires hostNetwork: true, configure pod anti-affinity to ensure that pods from the same replica set are scheduled to different nodes.

3. Verify that no other pod on the same node is using the port.

The pod log shows container init caused "setenv: invalid argument": unknown.

The workload mounts a Secret, but the value in the Secret is not Base64-encoded.

• Create the Secret in the console (values are auto-Base64-encoded). See Manage Secrets.

• Create the Secret from a YAML file and manually Base64-encode the value by running the echo -n "xxxxx" | base64 command.

Application-specific issue.

Examine the pod logs to troubleshoot the issue.

Error message

Description

Solution

The pod's events show Readiness probe failed: ....

The readiness probe failed, preventing the target pod from receiving traffic.

• Readiness probe configuration: On the Edit page of the target Workloads, verify that the health check path (for example, /healthz) and port match the application's. If the application starts slowly, increase the Unhealthy Threshold to avoid premature failures.

  Temporarily disable Readiness, then use curl from the pod terminal or host to verify the health check endpoint.

• Troubleshoot application issues: Investigate the issue by checking the pod's Events and Logs. Select Show the log of the last container exit.

The pod status is the same as above. The pod's events show Startup probe failed: ....

A failed startup probe causes the container to restart. This error should not result in a persistent Running/NotReady state but rather a 'CrashLoopBackOff' state.

Troubleshoot this issue as described in the "Pod fails to start (CrashLoopBackOff)" section for Startups.

OOM level

Description

Recommended solution

OS level

Check the kernel log at /var/log/messages on the pod's node. If the log shows a killed process but contains no cgroup logs, the OOM event occurred at the OS level.

• Scale up the node's memory resources or distribute the workload across more nodes. For more information, see Scale up or scale down node resources and Schedule applications to specific nodes.

• Identify Pods with high memory usage on the node and set appropriate memory limits for them.

cgroup level

Check the kernel log at /var/log/messages on the pod's node. If the log contains an error message similar to Task in /kubepods.slice/xxxxx killed as a result of limit of /kubepods.slice/xxxx, the OOM event occurred at the cgroup level.

• Increase the Pod's memory limits. As a best practice, ensure actual usage remains below 80% of the new limit. For more information, see Manage Pods and Scale up or scale down node resources.

• Enable resource profiling to get recommendations for container requests and limits.

Possible cause

Description

Recommended solution

The node is in the NotReady state.

The pod is automatically deleted after the node recovers from the NotReady state.

The pod is configured with finalizers.

If a pod is configured with finalizers, Kubernetes performs the cleanup operations specified by the finalizers before deleting the pod. If a cleanup operation fails to respond, the pod remains in the Terminating state.

Check the pod's finalizer configuration with kubectl get pod -n <ns> <pod name> -o yaml and investigate the cause.

The pod's preStop hook is invalid or stuck.

If a preStop hook is configured for the pod, Kubernetes executes the hook before terminating the container. The pod remains in the Terminating state while the hook is running.

Check the pod's preStop hook configuration with kubectl get pod -n <ns> <pod name> -o yaml and investigate the cause.

A graceful shutdown period is configured for the pod.

If a Pod is configured with a graceful shutdown period (terminationGracePeriodSeconds), the Pod enters the Terminating state after it receives a termination command, such as kubectl delete pod <pod_name>. Kubernetes considers the Pod to be successfully shut down only after the time specified in terminationGracePeriodSeconds elapses or the container exits.

Kubernetes automatically deletes the pod after the container completes a graceful shutdown.

The container is unresponsive.

When you request to stop or delete a pod, Kubernetes sends a SIGTERM signal to the containers in the pod. If a container does not correctly handle the SIGTERM signal during termination, the pod may remain in the Terminating state.

1. Forcefully delete the pod with kubectl delete pod <pod-name> --grace-period=0 --force.

2. Check the containerd or Docker logs on the pod's node to investigate further.

Possible cause

Description

Recommended solution

The node is under resource pressure from factors like memory or disk usage.

The node may be experiencing memory pressure, disk pressure, or PID pressure.

• Check node taints with kubectl describe node <node name> | grep Taints. The output may include:

  • Memory pressure: The node has the node.kubernetes.io/memory-pressure taint.

  • Disk pressure: The node has the node.kubernetes.io/disk-pressure taint.

  • PID pressure: The node has the node.kubernetes.io/pid-pressure taint.

• The pod status is one of the following:

  • Evicted

  • ContainerStatusUnknown, and the reason field in the pod's YAML file shows Evicted.

• Memory pressure:

  • Adjust the pod's resource configuration based on your business requirements. See Manage pods.

  • Upgrade the node. See Scale node resources.

• Disk pressure:

  • Periodically clear application logs from pods on the node to free up disk space.

  • Expand the node's disk. See Step 1: Resize a cloud disk.

• PID pressure: Adjust the pod's resource configuration. See Process ID Limits and Reservations.

An unexpected eviction occurs.

A manually added NoExecute taint on the pod's node caused an unexpected eviction.

Check for a NoExecute taint with kubectl describe node <node name> | grep Taints. If one exists, remove it.

Eviction does not proceed as expected.

• --pod-eviction-timeout: Pods on a failed node are evicted after this timeout period. The default is 5 minutes.

• --node-eviction-rate: The number of pods evicted from a node per second. The default is 0.1, meaning at most one pod is evicted from a node every 10 seconds.

• --secondary-node-eviction-rate: The secondary node eviction rate. If too many nodes in a cluster fail, the eviction rate is reduced to this value. The default is 0.01.

• --unhealthy-zone-threshold: The unhealthy availability zone threshold. The default is 0.55. When the fraction of failed nodes in an availability zone exceeds this threshold, the zone is considered unhealthy.

• --large-cluster-size-threshold: The large cluster size threshold. The default is 50. A cluster is considered large when it has more than 50 nodes.

In a small cluster (50 nodes or fewer), if more than 55% of nodes fail, pod eviction stops. See Rate limits on eviction.

In a large cluster (more than 50 nodes), if the fraction of unhealthy nodes exceeds the --unhealthy-zone-threshold (default 0.55), the eviction rate drops to --secondary-node-eviction-rate (default 0.01 pods per second). See Rate limits on eviction.

A pod is frequently rescheduled to its original node after being evicted.

The kubelet evicts pods based on actual resource usage, whereas the scheduler places pods based on resource requests. Because an eviction frees up resources, the scheduler might reschedule a pod to the same node if its requests still fit.

Adjust the pod's resource requests to fit the node's allocatable resources. See Set CPU and memory resources for a container. Enable resource profiling to get recommended request and limit values.