Troubleshoot DNS resolution errors

Test network connectivity between application pods and CoreDNS

Enter the application pod's network namespace using one of these methods:

• Method 1 (recommended): Run kubectl exec -it <pod-name> -- bash to enter the pod.

• Method 2: Log in to the node, find the process ID with ps aux | grep <application-process-name>, then enter the network namespace with nsenter -t <pid> -n bash.

• Method 3 (for frequently restarting pods):

  1. Log in to the node.

  2. Run docker ps -a | grep <application-container-name> to find sandboxed container IDs (names start with k8s_POD_).

  3. Run docker inspect <sandboxed-container-ID> | grep netns to find the network namespace path in /var/run/docker/netns/xxxx.

  4. Run nsenter -n bash to enter the namespace. > Note: Do not add a space between -n and <netns-path>.

From the pod's network namespace, test connectivity:

# Test connectivity to the kube-dns Service
dig <domain> @<kube-dns-svc-ip>

# Test Internet Control Message Protocol (ICMP) connectivity to the CoreDNS pod
ping <coredns-pod-ip>

# Test DNS query directly to the CoreDNS pod
dig <domain> @<coredns-pod-ip>

Replace <kube-dns-svc-ip> with the kube-dns Service IP in the kube-system namespace, and <coredns-pod-ip> with the IP of a CoreDNS pod.

Test network connectivity of CoreDNS

1. Log in to the node where the CoreDNS pod runs.

2. Run ps aux | grep coredns to get the CoreDNS process ID.

3. Run nsenter -t <pid> -n bash to enter the CoreDNS network namespace.

4. Test connectivity:

   # Test connectivity to the Kubernetes API server
   telnet <apiserver_clusterip> 6443  # apiserver_clusterip is the ClusterIP of the kubernetes Service in the default namespace.
   
   # Test connectivity to upstream DNS servers
   dig <domain> @100.100.2.136
   dig <domain> @100.100.2.138

Diagnose the container network

1. Log in to the ACK consoleACK console.

2. On the Clusters page, click the name of your cluster or click Details in the Actions column.

3. In the left-side navigation pane, choose Operations > Cluster Check.

4. On the Container Intelligence Service page, choose Cluster Check > Diagnosis.

5. On the Diagnosis page, click the Network Diagnosis tab.

6. Set Source address to the application pod IP, Destination address to the kube-dns Service IP, and Destination port to 53. Select Enable packet tracing and I know and agree, then click Create diagnosis.

7. In the diagnosis list, click Diagnosis details for your record.

The results show Diagnosis result, Packet paths, and All possible paths, along with identified error causes. See Use the cluster diagnostics feature to troubleshoot cluster issues.

Capture packets

Use packet capture when errors are intermittent and hard to reproduce.

1. Log in to the nodes where the application pods and the CoreDNS pod run.

2. Capture DNS traffic on each ECS instance:

   tcpdump -i any port 53 -C 20 -W 200 -w /tmp/client_dns.pcap

   This captures all traffic on port 53, rotating across up to 200 files of 20 MB each.

3. Reproduce the error and analyze packets from the failure time window. Check application logs for exact timestamps.

Packet capture has negligible service impact—only a slight increase in CPU utilization and disk I/O.

External domain name cannot be resolved

Check the CoreDNS query log for the response code. Enable the log plugin if not already on (see Configure DNS resolution), then search for the failing domain. NXDOMAIN means the domain does not exist upstream—often because a search suffix was appended, creating an invalid FQDN. SERVFAIL or REFUSED means the upstream server has a problem; check CoreDNS configuration and connectivity to 100.100.2.136 and 100.100.2.138.

Domain names of headless Services cannot be resolved

In CoreDNS before 1.7.0, API server network jitter can cause CoreDNS to exit, stopping headless Service record updates. Update to 1.7.0 or later. See \[Component Updates\] Update CoreDNS.

Domain names of StatefulSet pods cannot be resolved

The StatefulSet pod template must set serviceName to the headless Service name. Without this, per-pod DNS names (for example, pod.headless-svc.ns.svc.cluster.local) cannot be resolved, even though the Service-level name (for example, headless-svc.ns.svc.cluster.local) works. Set serviceName in the StatefulSet spec.

DNS queries are blocked by security group rules or network ACLs

Security group rules or network access control lists (ACLs) are blocking UDP port 53, causing DNS failures on affected nodes. Allow inbound and outbound UDP port 53 traffic.

Container network connectivity errors cause DNS failures

Container network errors block UDP port 53. Use the Network Diagnosis feature to identify the broken path and root cause.

CoreDNS pods are overloaded

When query volume exceeds CoreDNS replica capacity, latency rises and failures occur. Check whether CPU and memory usage is near the limit (kubectl -n kube-system top pod -l k8s-app=kube-dns).

Two fixes:

• Deploy NodeLocal DNSCache to absorb queries locally and reduce the load on CoreDNS. See Configure NodeLocal DNSCache.

• Scale out CoreDNS replicas so that peak CPU utilization per pod stays well below the node's available CPU.

DNS queries are not evenly distributed among CoreDNS pods

Imbalanced pod scheduling or a sessionAffinity setting on kube-dns can cause uneven query distribution. Symptom: noticeably different CPU utilization across CoreDNS pods.

Two fixes:

• Scale out CoreDNS pods and spread them across different nodes.

• Remove the sessionAffinity setting from the kube-dns Service. See Configure the kube-dns Service.

CoreDNS pods do not run as normal

Misconfigured YAML or ConfigMap can prevent CoreDNS from starting or cause crashes. Symptoms: pods not Running, increasing restart count, or log errors.

Check the CoreDNS log for these common errors:

DNS resolutions fail because the client is overloaded

When the ECS instance is fully loaded, UDP packets may drop before reaching CoreDNS. Look for abnormal network interface controller (NIC) retransmission rate and high CPU utilization in monitoring data.

Two options:

• Submit a ticketSubmit a ticketSubmit a ticketSubmit a ticketSubmit a ticketSubmit a ticketSubmit a ticketSubmit a ticketSubmit a ticketSubmit a ticket for detailed investigation.

• Deploy NodeLocal DNSCache to reduce inter-node DNS traffic. See Configure NodeLocal DNSCache.

Conntrack table is full

When the conntrack table is full, new UDP and TCP connections drop. This typically causes peak-hour DNS failures that recover off-peak. To confirm, run dmesg -H on the affected node and look for conntrack full during the failure window.

Increase the maximum number of entries in the conntrack table. See How do I increase the maximum number of tracked connections in the conntrack table of the Linux kernel?

autopath plugin does not work as normal

A known autopath plugin defect causes occasional resolution failures or wrong IPs for external domains. Internal domains resolve correctly. The issue worsens in clusters with high container creation rates.

Disable the autopath plugin:

1. Edit the CoreDNS ConfigMap with kubectl -n kube-system edit configmap coredns.

2. Delete the autopath @kubernetes line. Save and exit.

3. Verify the configuration loaded by checking CoreDNS logs for reload.

DNS resolutions fail due to concurrent A and AAAA record queries

Some Linux distributions send A and AAAA queries simultaneously over the same port, triggering conntrack conflicts that drop UDP packets.

Symptoms: intermittent resolution failures; packet capture shows simultaneous A and AAAA queries from the same source port.

Fixes depend on your image base:

• CentOS or Ubuntu: Add options timeout:2 attempts:3 rotate single-request-reopen to the DNS resolver configuration.

• Alpine Linux: Replace the Alpine-based image with one based on another OS. See Alpine caveats.

• PHP with cURL: Add CURL_IPRESOLVE_V4 to force IPv4-only resolution. See cURL functions.

• All environments: Deploy NodeLocal DNSCache, which mitigates the race condition. See Configure NodeLocal DNSCache.

DNS resolutions fail due to IPVS errors

In IPVS mode with CentOS or Alibaba Cloud Linux 2 (kernel before 4.19.91-25.1.al7.x86_64), removing UDP backend pods causes source port conflicts that drop packets. DNS failures last ~5 minutes during node or CoreDNS scaling events.

Two fixes:

• Deploy NodeLocal DNSCache to bypass the IPVS path for DNS queries. See Configure NodeLocal DNSCache.

• Shorten the UDP session timeout in IPVS mode. See Change the UDP timeout period in IPVS mode.

NodeLocal DNSCache does not work

DNS queries bypass NodeLocal DNSCache when either condition applies:

• dnsConfig was not injected into the application pods, so they still point to the kube-dns Service IP.

• Pods use an Alpine Linux base image, which queries all nameservers concurrently, including CoreDNS directly.

For the first case, enable automatic dnsConfig injection. See Configure NodeLocal DNSCache. For Alpine images, use an image built on another OS. See Alpine caveats.

Alibaba Cloud DNS PrivateZone names cannot be resolved

Alibaba Cloud DNS PrivateZone requires UDP, not TCP. With NodeLocal DNSCache, PrivateZone domains, vpc-proxy API endpoints, or other domain names may fail to resolve or resolve to wrong IPs.

Add prefer_udp to the CoreDNS configuration to force UDP for upstream queries. See Configure CoreDNS.