DNS best practices

Monitoring metrics

CoreDNS exposes health metrics, such as resolution results, through a standard Prometheus interface. This helps you detect anomalies in CoreDNS or upstream DNS servers.

Prometheus for ACK includes built-in CoreDNS monitoring metrics and alerting rules. You can enable the Prometheus and Dashboard features in the Container Service for Kubernetes console. For more information, see CoreDNS component monitoring.

If you run a self-managed Prometheus instance for your Kubernetes cluster, you can observe the relevant metrics and set alerts for critical ones. For more information, see the official CoreDNS Prometheus documentation.

Operational logs

When DNS anomalies occur, you can use CoreDNS logs to quickly diagnose the root causes. You can enable CoreDNS domain resolution logs and SLS log collection. For more information, see Analyze and monitor CoreDNS logs.

Kubernetes event delivery

In CoreDNS v1.9.3.6-32932850-aliyun and later, you can enable the k8s_event plugin to deliver critical CoreDNS logs as Kubernetes events to the Event Hub. For more information about the k8s_event plugin, see k8s_event.

Newly deployed CoreDNS instances have this feature enabled by default. If you are upgrading from an older version of CoreDNS, you must manually modify the configuration file to enable the feature.

1. Run the following command to open the CoreDNS configuration file.

   kubectl -n kube-system edit configmap/coredns

2. Add the kubeAPI and k8s_event plugins.

   apiVersion: v1
   data:
     Corefile: |
       .:53 {
           errors
           health {
               lameduck 15s
           }
           // Begin addition (ignore other differences).
           kubeapi
           k8s_event {
             level info error warning // Deliver critical logs with info, error, or warning levels.
           }
           // End addition.
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods verified
               fallthrough in-addr.arpa ip6.arpa
           }
           // Omitted below.
       }

3. Check the CoreDNS pod status and logs. If the logs contain the word reload, the modification is successful.

Assess CoreDNS component pressure

You can run DNS stress tests in your cluster to evaluate component pressure. Many open-source tools, such as DNSPerf, can help with this. If you cannot accurately assess the DNS pressure, follow these recommendations:

• Always deploy at least two CoreDNS pods, with each pod having resource limits of at least 1 CPU core and 1 GB of memory.

• The QPS capacity of CoreDNS is directly correlated with CPU usage. With NodeLocal DNSCache, each CPU core can support more than 10,000 QPS. The DNS QPS requirements of business workloads vary significantly. You should monitor the peak CPU usage of each CoreDNS pod. If the CPU usage exceeds one core during peak business periods, scale out the CoreDNS replicas. If the peak CPU usage is unknown, as a conservative measure, deploy one CoreDNS pod for every eight nodes in the cluster.

Adjust the number of CoreDNS pods

The number of CoreDNS pods directly determines the available compute resources. You can adjust this number based on your assessment.

• Automatically adjust the number of pods based on recommended policies

  You can deploy the following dns-autoscaler. It automatically adjusts the number of CoreDNS pods in real time based on the recommended ratio of one pod for every eight cluster nodes. The formula for the number of replicas is `replicas = max(ceil(cores × 1/coresPerReplica), ceil(nodes × 1/nodesPerReplica))`, which is constrained by the max and min limits.

  dns-autoscaler

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: dns-autoscaler
    namespace: kube-system
    labels:
      k8s-app: dns-autoscaler
  spec:
    selector:
      matchLabels:
        k8s-app: dns-autoscaler
    template:
      metadata:
        labels:
          k8s-app: dns-autoscaler
      spec:
        serviceAccountName: admin
        containers:
        - name: autoscaler
          image: registry.cn-hangzhou.aliyuncs.com/acs/cluster-proportional-autoscaler:1.8.4
          resources:
            requests:
              cpu: "200m"
              memory: "150Mi"
          command:
          - /cluster-proportional-autoscaler
          - --namespace=kube-system
          - --configmap=dns-autoscaler
          - --nodelabels=type!=virtual-kubelet
          - --target=Deployment/coredns
          - --default-params={"linear":{"coresPerReplica":64,"nodesPerReplica":8,"min":2,"max":100,"preventSinglePointFailure":true}}
          - --logtostderr=true
          - --v=9

• Manual adjustments

  You can manually adjust the number of CoreDNS pods by running the following command.

  kubectl scale --replicas={target} deployment/coredns -n kube-system # Replace {target} with the desired pod count

• Do not use workload autoscaling

  Although workload autoscaling mechanisms such as horizontal pod autoscaling (HPA) or CronHPA can automatically adjust the number of pods, they trigger frequent scaling operations. Due to the resolution anomalies that can occur during a pod scale-in as described earlier, do not use workload autoscaling to control the number of CoreDNS pods.

Adjust CoreDNS pod specifications

Another way to adjust CoreDNS resources is to modify the pod specifications. In ACK managed clusters Pro, CoreDNS pods have a default memory limit of 2Gi and no CPU limit. We recommend that you set the CPU limit to 4096m, with a minimum of 1024m. You can adjust the CoreDNS pod configuration in the console.

Schedule CoreDNS pods

Deploy CoreDNS pods across different zones and cluster nodes to avoid single-node or single-zone failures. CoreDNS versions earlier than v1.8.4.3 use weak node anti-affinity by default. This may cause some or all pods to be deployed on the same node due to insufficient node resources. If this occurs, you can delete the pods to trigger rescheduling or upgrade to the latest component version. CoreDNS versions earlier than v1.8 are no longer maintained. We recommend that you upgrade them as soon as possible.

Avoid deploying CoreDNS on cluster nodes that have full CPU or memory utilization because this affects DNS QPS and response latency. When possible, use custom parameters to schedule CoreDNS on dedicated cluster nodes to ensure stable domain name resolution.

Disable affinity settings for kube-dns service

Disable the Autopath plugin

Some older versions of CoreDNS have the Autopath plugin enabled. This plugin may produce incorrect resolution results in extreme scenarios. You can verify whether the plugin is enabled and disable it by editing the configuration file. For more information, see Autopath.

1. Run the kubectl -n kube-system edit configmap coredns command to open the CoreDNS configuration file.

2. Delete the autopath @kubernetes line and save the change.

3. Check the CoreDNS pod status and logs. If the logs contain the word reload, the modification is successful.

Configure graceful shutdown for CoreDNS

The lameduck mechanism in CoreDNS enables graceful shutdown. When CoreDNS stops or restarts, this mechanism ensures that ongoing requests are completed normally without being abruptly interrupted. The lameduck mechanism works as follows:

• When CoreDNS terminates, it enters Lameduck mode.

• In lameduck mode, CoreDNS stops accepting new requests but continues to process existing requests until they are complete or until the lameduck timeout period expires.

Set the default protocol for the Forward plugin to communicate with upstream VPC DNS servers

NodeLocal DNSCache communicates with CoreDNS using TCP. CoreDNS uses the same protocol as the incoming request when it communicates with upstream DNS servers. By default, external domain resolution requests from application containers pass through NodeLocal DNSCache and CoreDNS, and then reach the VPC DNS servers (100.100.2.136 and 100.100.2.138) using TCP.

# Before modification
forward . /etc/resolv.conf
# After modification
forward . /etc/resolv.conf {
  prefer_udp
}

Configure the Ready readiness probe plugin

CoreDNS versions later than 1.5.0 require the ready plugin to enable readiness probes.

1. Run the following command to open the CoreDNS configuration file.

   kubectl -n kube-system edit configmap/coredns

2. Check for the ready line. If it is missing, add ready. Press Esc, type :wq!, and then press Enter to save the change and exit.

   apiVersion: v1
   data:
    Corefile: |
     .:53 {
       errors
       health {
         lameduck 15s
       }
       ready # Add this line if missing, ensuring consistent indentation with Kubernetes.
       kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods verified
         fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf {
         max_concurrent 1000
               prefer_udp
       }
       cache 30
       loop
       log
       reload
       loadbalance
     }

3. Check the CoreDNS pod status and logs. If the logs contain the word reload, the modification is successful.

Configure the multisocket plugin to enhance CoreDNS resolution performance

CoreDNS v1.12.1 introduced the multisocket plugin. If you enable this plugin, CoreDNS can use multiple sockets to listen on the same port, which enhances performance in high-CPU scenarios. For more information about the plugin, see the community documentation.

You can enable multisocket in the coredns ConfigMap:

.:53 {
        ...
        prometheus :9153
        multisocket [NUM_SOCKETS]
        forward . /etc/resolv.conf
        ...
}

NUM_SOCKETS specifies the number of sockets that listen on the same port.

Configuration recommendations: We recommend that you align the value of NUM_SOCKETS with the estimated CPU usage, CPU resource limits, and available resources in the cluster. For example:

• If CoreDNS consumes 4 cores at its peak and 8 cores are available, set NUM_SOCKETS to 2.

• If CoreDNS consumes 8 cores at its peak and 64 cores are available, set NUM_SOCKETS to 8.

To determine the optimal configuration, you can test different settings and measure the QPS and load.

If you do not specify NUM_SOCKETS, CoreDNS uses GOMAXPROCS by default. The value of `GOMAXPROCS` is equal to the CPU limit of the CoreDNS pod. If no CPU limit is set, the value is equal to the number of CPU cores on the node.