If the Upgrade button is not displayed for the CoreDNS component on the Add-ons page in the Container Service for Kubernetes (ACK) console and you are using an older version of the component, your cluster does not support automatic CoreDNS upgrades. In this case, you must upgrade CoreDNS manually. This topic describes how to do so.
Prerequisites
Before you begin, ensure that you have:
-
kubectl configured to connect to your cluster. For more information, see Connect to a cluster using kubectl.
Before you begin
-
If your cluster uses IPVS in kube-proxy, you may experience DNS timeouts or failures cluster-wide for up to five minutes after the upgrade. To mitigate this issue, you can use one of the following methods:
-
Modify the UDP session persistence timeout for IPVS in kube-proxy. For more information, see Configure the UDP timeout for clusters that use IPVS below.
-
Use NodeLocal DNSCache. For more information, see Use NodeLocal DNSCache.
-
-
The upgrade takes about 2 minutes, but the actual time depends on the number of CoreDNS replicas in your cluster. During the upgrade, existing replicas are not stopped, ensuring that service resolution is not affected.
Check the current CoreDNS version
Console
Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click .
-
At the top of the Deployments page, set Namespace to kube-system. In the list of deployments, find the coredns deployment and check the Image column to confirm that the current CoreDNS image version is
coredns:1.6.2.
kubectl
Run the following command to check the current CoreDNS version:
kubectl get deployment coredns -n kube-system -o jsonpath="{.spec.template.spec.containers[0].image}"
The following is an example of the output:
registry-vpc.cn-hangzhou.aliyuncs.com/acs/coredns:1.6.2
Confirm the target upgrade version
Before upgrading, determine your target CoreDNS version. The following table shows the compatibility between CoreDNS and Kubernetes versions. We recommend using the latest compatible CoreDNS version for your cluster.
|
Kubernetes version |
CoreDNS version |
|
1.11 to 1.16 |
v1.6.2 (End of Life) |
|
1.14.8 to 1.22 |
v1.6.7 (End of Life), v1.7.0 |
|
1.20.4 and later |
v1.8.4, v1.9.3 Important
v1.8.4 and v1.9.3 have multiple specific versions, such as |
|
1.21 and later |
v1.11.3 |
Manually upgrade CoreDNS
Console
-
(Optional) Replace the
proxyplugin with theforwardplugin.The
proxyplugin was deprecated in CoreDNS v1.6.2. If you are upgrading from an earlier version, you must replace theproxyplugin with theforwardplugin in your CoreDNS configuration. -
Update the CoreDNS image version.
Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click .
-
On the Deployments page, set the Namespace to kube-system. Find coredns and in the Actions column, choose .
-
In the Edit YAML panel, find and update the
imagefield to your target version, and then click Update. In the YAML editor, find the CoreDNS deployment configuration and change the value of theimagefield to your target version, for example,v1.9.3.10-5e7ba42d-aliyun. The following is a key YAML snippet:labelSelector: matchExpressions: - key: k8s-app operator: In values: - kube-dns topologyKey: kubernetes.io/hostname weight: 100 containers: - args: - '-conf' - /etc/coredns/Corefile image: 'registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/coredns:v1.9.3.10-5e7ba42d-aliyun' imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 5 httpGet: ...
-
Verify that the upgrade is successful.
-
Run the following command to check the current CoreDNS version.
kubectl get deployment coredns -n kube-system -o jsonpath="{.spec.template.spec.containers[0].image}"Expected output:
registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/coredns:v1.9.3.10-5e7ba42d-aliyun -
Run the following command to check whether all CoreDNS pods in the cluster are in the Running state.
kubectl get pods -n kube-system | grep corednsExpected output:
coredns-78d4b8****-6g62w 1/1 Running 0 9d coredns-78d4b8****-n6wjm 1/1 Running 0 9d
-
kubectl
-
(Optional) Replace the
proxyplugin with theforwardplugin.The
proxyplugin was deprecated in CoreDNS v1.6.2. If you are upgrading from an earlier version, you must replace theproxyplugin with theforwardplugin in your CoreDNS configuration. -
Run the following command to edit the CoreDNS configuration, update the version in the
imagefield, and then save and exit.kubectl edit deployment/coredns -n kube-system -
Verify that the upgrade is successful.
-
Run the following command to check the current CoreDNS version.
kubectl get deployment coredns -n kube-system -o jsonpath="{.spec.template.spec.containers[0].image}"Expected output:
registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/coredns:v1.9.3.10-5e7ba42d-aliyun -
Run the following command to check whether all CoreDNS pods in the cluster are in the Running state.
kubectl get pods -n kube-system | grep corednsExpected output:
coredns-78d4b8****-6g62w 1/1 Running 0 9d coredns-78d4b8****-n6wjm 1/1 Running 0 9d
-
Configure UDP timeout for IPVS clusters
If your cluster uses kube-proxy in IPVS mode, the session persistence policy of IPVS can cause intermittent DNS resolution failures for up to five minutes after the upgrade. To minimize these failures, reduce the UDP session persistence timeout to 10 seconds. Before you proceed, evaluate the potential impact if your cluster runs UDP-based services.
If your cluster does not use IPVS mode, skip this section. To check the proxy mode of kube-proxy, see View cluster information.
Clusters that run Kubernetes 1.18 or later
Use the console
Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click .
-
On the ConfigMaps page, select the kube-system namespace, and then click Edit YAML in the Actions column for the kube-proxy-worker ConfigMap.
-
In the View in YAML panel, add
udpTimeout: 10sunder the ipvs field, and then click OK.apiVersion: v1 data: config.conf: | apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration # Other fields are omitted. mode: ipvs # If the ipvs key does not exist, add it. ipvs: udpTimeout: 10s -
Recreate all pods named kube-proxy-worker.
-
On the Cluster Information page, in the left navigation bar, select .
-
In the list of DaemonSets, find and click kube-proxy-worker.
-
On the kube-proxy-worker page, click the Pods tab. For each pod, choose in the Actions column and then click OK.
The system automatically recreates them.
-
-
Verify that the UDP timeout is configured.
-
Run the following command to install ipvsadm.
ipvsadm is a management tool for the IPVS module. For more information, see ipvsadm.
sudo yum install -y ipvsadm -
Run the following command on any ECS node in the cluster and check the third number in the output.
sudo ipvsadm -L --timeoutIf the third number is 10, the UDP timeout for the IPVS cluster has been changed successfully.
After the change is successful, wait at least five minutes before you proceed with the next step.
-
Use the command line
-
Run the following command to modify the kube-proxy-worker configuration file for kube-proxy.
kubectl -n kube-system edit configmap kube-proxy-worker -
In the kube-proxy configuration file, add
udpTimeout: 10sunder the ipvs field, and then save the file and exit.apiVersion: v1 data: config.conf: | apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration # Other fields are omitted. mode: ipvs # If the ipvs key does not exist, add it. ipvs: udpTimeout: 10s -
Run the following commands to recreate all pods named kube-proxy-worker.
-
Run the following command to view information about the existing pods.
kubectl -n kube-system get pod -o wide | grep kube-proxy-worker -
Run the following command to delete all pods returned in the previous step. The system automatically recreates the pods.
kubectl -n kube-system delete pod <kube-proxy-worker-****>Replace <kube-proxy-worker-****> with the name of each pod from the previous step.
-
-
Verify that the UDP timeout is configured.
-
Run the following command to install ipvsadm.
ipvsadm is a management tool for the IPVS module. For more information, see ipvsadm.
sudo yum install -y ipvsadm -
Run the following command on any ECS node in the cluster and check the third number in the output.
sudo ipvsadm -L --timeoutIf the third number is 10, the UDP timeout for the IPVS cluster has been changed successfully.
After the change is successful, wait at least five minutes before you proceed with the next step.
-
Clusters that run Kubernetes 1.16 or earlier
The kube-proxy component in clusters of this version does not support the udpTimeout parameter. You can use the OOS service to run the ipvsadm command on all nodes in the cluster to adjust the UDP timeout configuration. The command is as follows:
sudo yum install -y ipvsadm
sudo ipvsadm -L --timeout > /tmp/ipvsadm_timeout_old
sudo ipvsadm --set 900 120 10
sudo ipvsadm -L --timeout > /tmp/ipvsadm_timeout_new
diff /tmp/ipvsadm_timeout_old /tmp/ipvsadm_timeout_new
To learn how to run commands on multiple instances in a batch by using OOS, see Run commands on multiple instances in a batch.
Next steps
After the upgrade is complete, you can optimize the CoreDNS configuration based on your business requirements. For more information, see Optimize CoreDNS configurations.