DNS best practices

Metrics

CoreDNS exposes health metrics, including resolution results, through a standard Prometheus interface. This helps detect anomalies on the CoreDNS server and even upstream DNS servers.

Managed Service for Prometheus provides built-in metrics monitoring dashboards and alerting rules for CoreDNS. You can enable Prometheus and its dashboard features in the ACK console. For more information, see Monitor the CoreDNS component.

If you use a self-managed Prometheus instance to monitor your Kubernetes cluster, you can observe the relevant metrics in Prometheus and set up alerts for key indicators. For more information, see the official CoreDNS documentation for Prometheus.

Logs

In the event of a DNS anomaly, CoreDNS logs can help you quickly diagnose the root cause. We recommend that you enable CoreDNS domain name resolution logging and collect its logs with Log Service. For more information, see Analyze and monitor CoreDNS logs.

Kubernetes event delivery

In CoreDNS v1.9.3.6-32932850-aliyun and later, you can enable the k8s_event plugin to deliver critical CoreDNS logs as Kubernetes events to the Event Center. For more information about the k8s_event plugin, see k8s_event.

This feature is enabled by default in new CoreDNS deployments. If you upgrade from an earlier version to CoreDNS v1.9.3.6-32932850-aliyun or later, you need to manually modify the configuration file to enable it.

1. Run the following command to open the CoreDNS configuration file.

   kubectl -n kube-system edit configmap/coredns

2. Add the kubeapi and k8s_event plugins.

   apiVersion: v1
   data:
     Corefile: |
       .:53 {
           errors
           health {
               lameduck 15s
           }
           // Start of addition (ignore other differences).
           kubeapi
           k8s_event {
             level info error warning // Deliver critical logs with info, error, and warning statuses.
           }
           // End of addition.
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods verified
               fallthrough in-addr.arpa ip6.arpa
           }
           // ... (remaining content omitted)
       }

3. Check the operational status and logs of the CoreDNS pods. If the logs contain the word reload, the modification is successful.

Assess CoreDNS component pressure

You can perform a DNS stress test in the cluster to assess component pressure. Many open-source tools, including DNSPerf, can help you with this. If you cannot accurately assess the DNS pressure in your cluster, follow these recommendations.

• Always set the number of CoreDNS pods to at least 2, with a resource limit of at least 1 core and 1 GiB for a single pod.

• CoreDNS's domain name resolution QPS is positively correlated with its CPU consumption. With NodeLocal DNSCache enabled, each CPU core can support over 10,000 QPS. The QPS demand for domain name requests varies significantly across different types of services. You can observe the peak CPU usage of each CoreDNS pod. If a pod uses more than one CPU core during peak hours, we recommend that you scale out the CoreDNS replicas. If you cannot determine the peak CPU usage, you can conservatively use a 1:8 ratio of pods to cluster nodes. That is, for every 8 cluster nodes that you add, add one CoreDNS pod.

Adjust CoreDNS pod count

The number of CoreDNS pods directly determines the computing resources that CoreDNS can use. You can adjust the number of CoreDNS pods based on your assessment.

• Automatically adjust based on the recommended policy

  You can deploy the following dns-autoscaler. It automatically adjusts the number of CoreDNS pods in real time based on the recommended policy (a 1:8 ratio of pods to cluster nodes). The number of pods is calculated by using the following formula: replicas = max(ceil(cores × 1/coresPerReplica), ceil(nodes × 1/nodesPerReplica)), and is limited by the max and min parameters.

  dns-autoscaler

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: dns-autoscaler
    namespace: kube-system
    labels:
      k8s-app: dns-autoscaler
  spec:
    selector:
      matchLabels:
        k8s-app: dns-autoscaler
    template:
      metadata:
        labels:
          k8s-app: dns-autoscaler
      spec:
        serviceAccountName: admin
        containers:
        - name: autoscaler
          image: registry.cn-hangzhou.aliyuncs.com/acs/cluster-proportional-autoscaler:1.8.4
          resources:
            requests:
              cpu: "200m"
              memory: "150Mi"
          command:
          - /cluster-proportional-autoscaler
          - --namespace=kube-system
          - --configmap=dns-autoscaler
          - --nodelabels=type!=virtual-kubelet
          - --target=Deployment/coredns
          - --default-params={"linear":{"coresPerReplica":64,"nodesPerReplica":8,"min":2,"max":100,"preventSinglePointFailure":true}}
          - --logtostderr=true
          - --v=9

• Manually adjust

  You can run the following command to manually adjust the number of CoreDNS pods.

  kubectl scale --replicas={target} deployment/coredns -n kube-system # Replace {target} with the desired number of pods.

• Do not use workload auto-scaling

  Although workload auto-scaling features like Horizontal Pod Autoscaler (HPA) and CronHPA can also automatically adjust the number of pods, they perform frequent scaling operations. Due to the resolution exceptions that occur when pods are scaled in, do not use workload auto-scaling to control the number of CoreDNS pods.

Adjust CoreDNS pod specifications

Another way to adjust CoreDNS resources is to modify pod specifications. In an ACK managed Pro cluster, the default memory limit for CoreDNS pods is 2Gi, with no CPU limit. Set the CPU limit to 4096m, with a minimum of 1024m. You can adjust the CoreDNS pod configuration in the console.

Schedule CoreDNS pods

We recommend that you deploy CoreDNS pods across different availability zones and cluster nodes to avoid single-node or single-availability-zone failures. CoreDNS component versions earlier than v1.8.4.3 have a default soft anti-affinity policy at the node level, which may cause some or all pods to be deployed on the same node if resources are insufficient. If this occurs, delete the pods to trigger rescheduling, or upgrade the component to the latest version. CoreDNS component versions earlier than v1.8 are no longer maintained and should be upgraded as soon as possible.

The cluster nodes where CoreDNS runs should not have their CPU or memory fully utilized, because this affects the QPS and response latency of domain name resolution. When cluster node conditions permit, consider using custom parameters to schedule CoreDNS to dedicated cluster nodes to provide a stable domain name resolution service.

Disable session affinity for kube-dns

Disable the autopath plugin

Some earlier versions of CoreDNS enabled the autopath plugin, which can cause resolution errors in some edge cases. Check if it is enabled and edit the configuration file to disable it. For more information, see Autopath.

1. Run the kubectl -n kube-system edit configmap coredns command to open the CoreDNS configuration file.

2. Delete the autopath @kubernetes line and save the file.

3. Check the operational status and logs of the CoreDNS pods. If the logs contain the word reload, the modification is successful.

Configure graceful shutdown

lameduck is a mechanism in CoreDNS that enables graceful shutdown. It ensures that when CoreDNS needs to stop or restart, ongoing requests are completed without being abruptly interrupted. lameduck works as follows:

• When a CoreDNS process is about to terminate, it enters Lameduck mode.

• In lameduck mode, CoreDNS stops accepting new requests but continues to process existing requests until they are all completed or the lameduck timeout period is exceeded.

Configure default protocol for the forward plugin

NodeLocal DNSCache uses TCP to communicate with CoreDNS. CoreDNS then communicates with upstream DNS servers by using the same protocol as the incoming request. Therefore, by default, requests from business containers to resolve domain names outside the cluster pass through NodeLocal DNSCache and CoreDNS, and finally reach the VPC DNS servers (by default, 100.100.2.136 and 100.100.2.138 on ECS instances) over TCP.

# Before modification
forward . /etc/resolv.conf
# After modification
forward . /etc/resolv.conf {
  prefer_udp
}

Configure the ready plugin

CoreDNS versions later than 1.5.0 must have the ready plugin configured to enable readiness probes.

1. Run the following command to open the CoreDNS configuration file.

   kubectl -n kube-system edit configmap/coredns

2. Check if the file contains the ready line. If not, add the ready line, press Esc, enter :wq!, and then press Enter to save the modified configuration file and exit edit mode.

   apiVersion: v1
   data:
    Corefile: |
     .:53 {
       errors
       health {
         lameduck 15s
       }
       ready # If this line does not exist, add it. Make sure that the indentation is consistent with Kubernetes.
       kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods verified
         fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf {
         max_concurrent 1000
               prefer_udp
       }
       cache 30
       loop
       log
       reload
       loadbalance
     }

3. Check the operational status and logs of the CoreDNS pods. If the logs contain the word reload, the modification is successful.

Enhance performance with the multisocket plugin

CoreDNS v1.12.1 introduced the multisocket plugin. Enabling this plugin allows CoreDNS to use multiple sockets to listen on the same port, enhancing CoreDNS performance in high-CPU scenarios. For a detailed description of the plugin, see the community documentation.

You need to enable multisocket by using the coredns ConfigMap:

.:53 {
        ...
        prometheus :9153
        multisocket [NUM_SOCKETS]
        forward . /etc/resolv.conf
        ...
}

NUM_SOCKETS specifies the number of sockets that listen on the same port.

Recommended configuration: Align NUM_SOCKETS with the estimated CPU utilization, CPU resource limits, and available cluster resources. For example:

• If CoreDNS consumes 4 cores at peak and 8 cores are available, set NUM_SOCKETS to 2.

• If CoreDNS consumes 8 cores at peak and 64 cores are available, set NUM_SOCKETS to 8.

To determine the optimal configuration, we recommend that you test the QPS and load with different settings.

If you do not specify NUM_SOCKETS, the default value is GOMAXPROCS, which is equal to the CPU limit of the CoreDNS pod. If the pod's CPU limit is not set, the value is equal to the number of CPU cores on the node where the pod is running.