Manually upgrade unmanaged CoreDNS

更新时间:
复制 MD 格式

If the Upgrade button is not displayed for the CoreDNS component on the Add-ons page in the Container Service for Kubernetes (ACK) console and you are using an older version of the component, your cluster does not support automatic CoreDNS upgrades. In this case, you must upgrade CoreDNS manually. This topic describes how to do so.

Prerequisites

Before you begin, ensure that you have:

Before you begin

  • If your cluster uses IPVS in kube-proxy, you may experience DNS timeouts or failures cluster-wide for up to five minutes after the upgrade. To mitigate this issue, you can use one of the following methods:

  • The upgrade takes about 2 minutes, but the actual time depends on the number of CoreDNS replicas in your cluster. During the upgrade, existing replicas are not stopped, ensuring that service resolution is not affected.

Check the current CoreDNS version

Console

  1. Log on to the ACK console. In the left navigation pane, click Clusters.

  2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > Deployments.

  3. At the top of the Deployments page, set Namespace to kube-system. In the list of deployments, find the coredns deployment and check the Image column to confirm that the current CoreDNS image version is coredns:1.6.2.

kubectl

Run the following command to check the current CoreDNS version:

kubectl get deployment coredns -n kube-system -o jsonpath="{.spec.template.spec.containers[0].image}"

The following is an example of the output:

registry-vpc.cn-hangzhou.aliyuncs.com/acs/coredns:1.6.2

Confirm the target upgrade version

Before upgrading, determine your target CoreDNS version. The following table shows the compatibility between CoreDNS and Kubernetes versions. We recommend using the latest compatible CoreDNS version for your cluster.

Kubernetes version

CoreDNS version

1.11 to 1.16

v1.6.2 (End of Life)

1.14.8 to 1.22

v1.6.7 (End of Life), v1.7.0

1.20.4 and later

v1.8.4, v1.9.3

Important

v1.8.4 and v1.9.3 have multiple specific versions, such as v1.8.4.3-644f4735-aliyun and v1.9.3.1-5e7ba42d-aliyun. For more information, see CoreDNS.

1.21 and later

v1.11.3

Manually upgrade CoreDNS

Console

  1. (Optional) Replace the proxy plugin with the forward plugin.

    The proxy plugin was deprecated in CoreDNS v1.6.2. If you are upgrading from an earlier version, you must replace the proxy plugin with the forward plugin in your CoreDNS configuration.

    Change the CoreDNS configuration

    1. Log on to the ACK console. In the left navigation pane, click Clusters.

    2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Configurations > ConfigMaps.

    3. On the ConfigMaps page, set Namespace to kube-system. Find coredns and click Edit YAML in the Actions column.

    4. In the View in YAML panel, change proxy to forward, and then click OK. This replaces the deprecated proxy plugin with the forward plugin. The final ConfigMap should look similar to the following:

      apiVersion: v1
      data:
        Corefile: |-
          .:53 {
            autopath @kubernetes
            cache 30
            errors
            forward . /etc/resolv.conf
            health
            kubernetes cluster.local in-addr.arpa ip6.arpa {
                pods verified
                fallthrough in-addr.arpa ip6.arpa
            }
            loadbalance
            loop
            prometheus :9153
            ready
            reload
          }
      kind: ConfigMap
  2. Update the CoreDNS image version.

    1. Log on to the ACK console. In the left navigation pane, click Clusters.

    2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > Deployments.

    3. On the Deployments page, set the Namespace to kube-system. Find coredns and in the Actions column, choose image.png > Edit YAML.

    4. In the Edit YAML panel, find and update the image field to your target version, and then click Update. In the YAML editor, find the CoreDNS deployment configuration and change the value of the image field to your target version, for example, v1.9.3.10-5e7ba42d-aliyun. The following is a key YAML snippet:

      labelSelector:
                matchExpressions:
                  - key: k8s-app
                    operator: In
                    values:
                      - kube-dns
              topologyKey: kubernetes.io/hostname
            weight: 100
        containers:
        - args:
          - '-conf'
          - /etc/coredns/Corefile
          image: 'registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/coredns:v1.9.3.10-5e7ba42d-aliyun'
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 5
            httpGet:
              ...
  3. Verify that the upgrade is successful.

    1. Run the following command to check the current CoreDNS version.

      kubectl get deployment coredns -n kube-system -o jsonpath="{.spec.template.spec.containers[0].image}"

      Expected output:

      registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/coredns:v1.9.3.10-5e7ba42d-aliyun
    2. Run the following command to check whether all CoreDNS pods in the cluster are in the Running state.

      kubectl get pods -n kube-system | grep coredns

      Expected output:

      coredns-78d4b8****-6g62w                           1/1     Running   0          9d
      coredns-78d4b8****-n6wjm                           1/1     Running   0          9d

kubectl

  1. (Optional) Replace the proxy plugin with the forward plugin.

    The proxy plugin was deprecated in CoreDNS v1.6.2. If you are upgrading from an earlier version, you must replace the proxy plugin with the forward plugin in your CoreDNS configuration.

    Change the CoreDNS configuration

    1. Run the following command to edit the CoreDNS configuration file, change proxy to forward, and then save the file and exit.

      kubectl edit configmap/coredns -n kube-system
    2. Run the following command to view the logs of a CoreDNS pod and confirm that the configuration is reloaded.

      kubectl logs coredns-78d4b8bd88-n6wjm -n kube-system

      The expected output is as follows. The presence of plugin/reload indicates that the CoreDNS configuration has been reloaded successfully.

      .:53
      [INFO] plugin/reload: Running configuration MD5 = 71c5f1ff539d304c630521f315dc2ac2
      CoreDNS-1.6.7
      linux/amd64, go1.13.6, da7f65b
      [INFO] 127.0.0.1:48329 - 42313 "HINFO IN 1108347002237365533.4506541768939609094. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.008874794s
  2. Run the following command to edit the CoreDNS configuration, update the version in the image field, and then save and exit.

    kubectl edit deployment/coredns -n kube-system
  3. Verify that the upgrade is successful.

    1. Run the following command to check the current CoreDNS version.

      kubectl get deployment coredns -n kube-system -o jsonpath="{.spec.template.spec.containers[0].image}"

      Expected output:

      registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/coredns:v1.9.3.10-5e7ba42d-aliyun
    2. Run the following command to check whether all CoreDNS pods in the cluster are in the Running state.

      kubectl get pods -n kube-system | grep coredns

      Expected output:

      coredns-78d4b8****-6g62w                           1/1     Running   0          9d
      coredns-78d4b8****-n6wjm                           1/1     Running   0          9d

Configure UDP timeout for IPVS clusters

If your cluster uses kube-proxy in IPVS mode, the session persistence policy of IPVS can cause intermittent DNS resolution failures for up to five minutes after the upgrade. To minimize these failures, reduce the UDP session persistence timeout to 10 seconds. Before you proceed, evaluate the potential impact if your cluster runs UDP-based services.

If your cluster does not use IPVS mode, skip this section. To check the proxy mode of kube-proxy, see View cluster information.

Clusters that run Kubernetes 1.18 or later

Use the console

  1. Log on to the ACK console. In the left navigation pane, click Clusters.

  2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Configurations > ConfigMaps.

  3. On the ConfigMaps page, select the kube-system namespace, and then click Edit YAML in the Actions column for the kube-proxy-worker ConfigMap.

  4. In the View in YAML panel, add udpTimeout: 10s under the ipvs field, and then click OK.

    apiVersion: v1
    data:
      config.conf: |
        apiVersion: kubeproxy.config.k8s.io/v1alpha1
        kind: KubeProxyConfiguration
        # Other fields are omitted.
        mode: ipvs
        # If the ipvs key does not exist, add it.
        ipvs:
          udpTimeout: 10s
  5. Recreate all pods named kube-proxy-worker.

    1. On the Cluster Information page, in the left navigation bar, select Workloads > DaemonSets.

    2. In the list of DaemonSets, find and click kube-proxy-worker.

    3. On the kube-proxy-worker page, click the Pods tab. For each pod, choose More > Delete in the Actions column and then click OK.

      The system automatically recreates them.

  6. Verify that the UDP timeout is configured.

    1. Run the following command to install ipvsadm.

      ipvsadm is a management tool for the IPVS module. For more information, see ipvsadm.

      sudo yum install -y ipvsadm
    2. Run the following command on any ECS node in the cluster and check the third number in the output.

      sudo ipvsadm -L --timeout

      If the third number is 10, the UDP timeout for the IPVS cluster has been changed successfully.

      After the change is successful, wait at least five minutes before you proceed with the next step.

Use the command line

  1. Run the following command to modify the kube-proxy-worker configuration file for kube-proxy.

    kubectl -n kube-system edit configmap kube-proxy-worker
  2. In the kube-proxy configuration file, add udpTimeout: 10s under the ipvs field, and then save the file and exit.

    apiVersion: v1
    data:
      config.conf: |
        apiVersion: kubeproxy.config.k8s.io/v1alpha1
        kind: KubeProxyConfiguration
        # Other fields are omitted.
        mode: ipvs
        # If the ipvs key does not exist, add it.
        ipvs:
          udpTimeout: 10s
  3. Run the following commands to recreate all pods named kube-proxy-worker.

    1. Run the following command to view information about the existing pods.

      kubectl -n kube-system get pod -o wide | grep kube-proxy-worker
    2. Run the following command to delete all pods returned in the previous step. The system automatically recreates the pods.

      kubectl -n kube-system delete pod <kube-proxy-worker-****>
      Replace <kube-proxy-worker-****> with the name of each pod from the previous step.
  4. Verify that the UDP timeout is configured.

    1. Run the following command to install ipvsadm.

      ipvsadm is a management tool for the IPVS module. For more information, see ipvsadm.

      sudo yum install -y ipvsadm
    2. Run the following command on any ECS node in the cluster and check the third number in the output.

      sudo ipvsadm -L --timeout

      If the third number is 10, the UDP timeout for the IPVS cluster has been changed successfully.

      After the change is successful, wait at least five minutes before you proceed with the next step.

Clusters that run Kubernetes 1.16 or earlier

The kube-proxy component in clusters of this version does not support the udpTimeout parameter. You can use the OOS service to run the ipvsadm command on all nodes in the cluster to adjust the UDP timeout configuration. The command is as follows:

sudo yum install -y ipvsadm
sudo ipvsadm -L --timeout > /tmp/ipvsadm_timeout_old
sudo ipvsadm --set 900 120 10
sudo ipvsadm -L --timeout > /tmp/ipvsadm_timeout_new
diff /tmp/ipvsadm_timeout_old /tmp/ipvsadm_timeout_new

To learn how to run commands on multiple instances in a batch by using OOS, see Run commands on multiple instances in a batch.

Next steps

After the upgrade is complete, you can optimize the CoreDNS configuration based on your business requirements. For more information, see Optimize CoreDNS configurations.