Container networking FAQ

Index

Terway network modes

Terway supports shared ENI mode and exclusive ENI mode. For details about each mode, see Shared ENI mode and exclusive ENI mode. Only Terway shared ENI mode supports network acceleration (DataPathv2 or IPvlan+eBPF). DataPathv2 is an upgrade to the earlier IPvlan+eBPF acceleration mode. With Terway v1.8.0 or later, you can only select DataPathv2 for acceleration when creating a cluster and installing the Terway plugin.

Differentiate between exclusive and shared ENI modes

• In Terway v1.11.0 and later, Terway uses shared ENI mode by default. You can enable exclusive ENI mode by configuring the exclusive ENI network mode for a node pool.

• For Terway versions earlier than v1.11.0, you can select either exclusive ENI mode or shared ENI mode during cluster creation. After you create a cluster, you can identify the mode as follows:

  • Exclusive ENI mode: The Terway DaemonSet in the kube-system namespace is named terway-eni.

  • Shared ENI mode: The Terway DaemonSet in the kube-system namespace is named terway-eniip.

Determine the network acceleration mode

Only Terway shared ENI mode supports network acceleration (DataPathv2 or IPvlan+eBPF). DataPathv2 is an upgraded version of the legacy IPvlan+eBPF acceleration mode. In Terway v1.8.0 and later, you can select only DataPathv2 for acceleration when you create a cluster and install the Terway plugin.

To determine which network acceleration mode is active, check the eniip_virtual_type setting in the eni-config ConfigMap in the kube-system namespace. The value is either datapathv2 (indicating DataPathv2) or ipvlan (indicating IPvlan+eBPF).

IPVS bypass in acceleration mode

In acceleration mode (DataPathv2 or IPvlan+eBPF), Terway uses a different traffic forwarding path from standard shared ENI mode. For example, when a Pod accesses an internal Service, traffic bypasses the node's network stack and IPVS routing. Terway uses eBPF to resolve a Service address to a backend Pod's address. For details on traffic flow, see Network acceleration.

Switching network plugins

No. The network plugin, either Terway or Flannel, can only be selected during cluster creation and cannot be changed afterward. To switch plugins, you must create a new cluster. For more information, see Create an ACK managed cluster.

Cluster cannot access the internet after adding a vSwitch in Terway

Symptom

You manually add a vSwitch to a Terway network to resolve an IP address shortage for Pods. After adding the vSwitch, the cluster cannot access the internet.

Cause

The vSwitch that provides IP addresses to the Pods is not configured for internet access.

Solution

Use a NAT Gateway to configure an SNAT rule for the vSwitch. For more information, see Enable internet access for a cluster.

Resolve Flannel incompatibility on Kubernetes 1.16+

Symptom

After upgrading a cluster to Kubernetes 1.16 or later, the nodes become NotReady.

Cause

Because the Flannel version was manually upgraded but its configuration was not, Kubelet is unable to recognize the configuration.

Solution

1. Edit the Flannel ConfigMap to add the cniVersion field.

   kubectl edit cm kube-flannel-cfg -n kube-system 

   The cniVersion field is added to the response.

   "name": "cb0",   
   "cniVersion":"0.3.0",
   "type": "flannel",

2. Restart the Flannel pods to apply the new configuration.

   kubectl delete pod -n kube-system -l app=flannel

Pod startup latency

Symptom

A Pod experiences network latency for a short period after it starts.

Cause

Enforcing a network policy can introduce latency.

Solution

1. Edit the Terway ConfigMap to disable network policy.

   kubectl edit cm -n kube-system eni-config

   Add the following field to the configuration:

   disable_network_policy: "true"

2. Optional: If you are using an earlier version of Terway, upgrade the Terway add-on on the console.

   1. Log on to the ACK console. In the left navigation pane, click Clusters.

   2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Add-ons.

   3. On the Add-ons page, click the Networking tab, and then click Upgrade for the Terway add-on.

   4. In the dialog box, complete the configuration and click OK.

3. Restart all Terway Pods.

   kubectl delete pod -n kube-system -l app=terway-eniip

Enable hairpinning for Pods

Symptom

A Pod fails to access a service it exposes. You might experience intermittent connection failures, or the connection fails when traffic is routed back to the same Pod.

Cause

Solution

• Use a Headless Service to expose and access the service. For more information, see Headless Services.

  Note

  This is the recommended method.

• Recreate the cluster and use the Terway network plugin. For more information, see Use the Terway network plugin.

• Modify the Flannel configuration, and then recreate the Flannel Pods and your application Pods.

  Note

  This method is not recommended because future upgrades might overwrite the configuration.

  1. Edit cni-config.json.

     kubectl edit cm kube-flannel-cfg -n kube-system

  2. In the returned configuration, add hairpinMode: true to the delegate section.

     Example:

     cni-conf.json: |
         {
           "name": "cb0",
           "cniVersion":"0.3.1",
           "type": "flannel",
           "delegate": {
             "isDefaultGateway": true,
             "hairpinMode": true
           }
         }

  3. Restart the Flannel Pods.

     kubectl delete pod -n kube-system -l app=flannel   

  4. Delete and recreate your application Pods.

Choose between Terway and Flannel network plugins

This topic helps you choose between Terway and Flannel, the two network plugins available for ACK clusters.

Cluster network planning

To create an ACK cluster, you must specify a VPC, vSwitches, a pod CIDR block, and a service CIDR block. We recommend planning the address spaces for your ECS instances, pods, and services. For more information, see Plan the network for an ACK managed cluster.

Support for hostPort mapping

• Only the Flannel plugin supports hostPort; other plugins currently do not.

• Pod IP addresses in ACK are directly accessible by other resources within the same VPC, so no additional port mapping is required.

• To expose a service externally, use a NodePort or LoadBalancer service.

View cluster network type and vSwitches

ACK supports two container network types: Flannel and Terway.

• Follow these steps to view the cluster's network type.

  1. Log on to the ACK console. In the left navigation pane, click Clusters.

  2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Cluster Information.

  3. Click the Basic Information tab. In the Network section, the Network Plug-in field shows the container network type.

     • If Terway is displayed to the right of Network Plug-in, the container network type is Terway network.

     • If Flannel is displayed to the right of Network Plug-in, the container network type is Flannel network.

• Follow these steps to view the vSwitches for the cluster nodes.

  1. In the left-side navigation pane, choose Nodes > Node Pools.

  2. On the Node Pools page, click Details in the Actions column of the target node pool, and then click the Basic Information tab.

     In the Node configuration section, find the Node vSwitch ID.

• Follow these steps to find the Pod vSwitch ID for Terway networks.

  Note

  Only Terway networks require a Pod vSwitch.

  1. In the left-side navigation pane, click Add-ons.

  2. On the Add-ons page, click Configuration on the terway-eniip card. The PodVswitchId field shows the current Pod vSwitch ID.

View cloud resources

Modify the kube-proxy configuration

By default, an ACK managed cluster uses the kube-proxy-worker DaemonSet as its load balancer. You can control its parameters with the kube-proxy-worker ConfigMap. If you are using an ACK dedicated cluster, an additional kube-proxy-master DaemonSet and its corresponding ConfigMap run on the control plane nodes.

The kube-proxy ConfigMaps are compatible with the community KubeProxyConfiguration standard. You can customize the configuration according to this standard. For more information, see kube-proxy Configuration. The kube-proxy configuration file requires strict formatting. Do not omit any colons or spaces. To modify the kube-proxy configuration:

• If you are using an ACK managed cluster, modify the kube-proxy-worker configuration.

  1. Log on to the ACK console. In the left navigation pane, click Clusters.

  2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Configurations > ConfigMaps.

  3. Select the kube-system namespace from the drop-down list at the top of the page. Then, click Edit YAML to the right of the kube-proxy-worker ConfigMap.

  4. In the View in YAML panel, modify the parameters and then click OK.

  5. Recreate all kube-proxy-worker pods to apply the new configuration.

     Important

     Restarting kube-proxy does not interrupt existing services. However, a brief delay may occur while programming new rules for concurrently deployed services. We recommend that you perform this operation during off-peak hours.

     1. In the left-side navigation pane, choose Workloads > DaemonSets.

     2. In the DaemonSet list, find and click kube-proxy-worker.

     3. On the kube-proxy-worker page, on the Pods tab, choose More > Delete and click OK.

        Repeat this step to delete all pods. After you delete the pods, the system automatically recreates them.

• For an ACK dedicated cluster, modify both the kube-proxy-worker and kube-proxy-master configurations, then delete their pods. The system automatically recreates the pods with the new configuration. The procedure is the same as described above.

Increase the Linux conntrack limit

Modify IPVS load balancing mode in kube-proxy

Modify the IPVS UDP session timeout

If your ACK cluster uses kube-proxy in IPVS mode, the default session persistence policy can cause intermittent packet loss for up to five minutes after a UDP backend is removed. If your services depend on CoreDNS, you might experience service latency and request timeouts for five minutes when the CoreDNS component is upgraded or its node is restarted.

Troubleshooting IPv6 dual-stack issues

• Problem: The Pod IP displayed in kubectl is still an IPv4 address.

  Solution: Run the following command to display the podIPs field. The output should include an IPv6 address.

  kubectl get pods -A -o jsonpath='{range .items[*]}{@.metadata.namespace} {@.metadata.name} {@.status.podIPs[*].ip} {"\n"}{end}'

• Problem: The cluster IP displayed in kubectl is still an IPv4 address.

  Solution:

  1. Verify that the spec.ipFamilyPolicy of the Service is not set to SingleStack.

  2. Run the following command to display the clusterIPs field. The output should include an IPv6 address.

     kubectl get svc -A -o jsonpath='{range .items[*]}{@.metadata.namespace} {@.metadata.name} {@.spec.ipFamilyPolicy} {@.spec.clusterIPs[*]} {"\n"}{end}'

• Problem: The Pod cannot be accessed using its IPv6 address.

  Cause: Some applications, such as Nginx containers, do not listen on IPv6 addresses by default.

  Solution: Run the netstat -anp command to verify that the Pod is listening on an IPv6 address.

  Expected output:

  Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
  tcp        0      0 127.0.XX.XX:10248         0.0.0.0:*               LISTEN      8196/kubelet
  tcp        0      0 127.0.XX.XX:41935         0.0.0.0:*               LISTEN      8196/kubelet
  tcp        0      0 0.0.XX.XX:111             0.0.0.0:*               LISTEN      598/rpcbind
  tcp        0      0 0.0.XX.XX:22              0.0.0.0:*               LISTEN      3577/sshd
  tcp6       0      0 :::30500                :::*                    LISTEN      1916680/kube-proxy
  tcp6       0      0 :::10250                :::*                    LISTEN      8196/kubelet
  tcp6       0      0 :::31183                :::*                    LISTEN      1916680/kube-proxy
  tcp6       0      0 :::10255                :::*                    LISTEN      8196/kubelet
  tcp6       0      0 :::111                  :::*                    LISTEN      598/rpcbind
  tcp6       0      0 :::10256                :::*                    LISTEN      1916680/kube-proxy
  tcp6       0      0 :::31641                :::*                    LISTEN      1916680/kube-proxy
  udp        0      0 0.0.0.0:68              0.0.0.0:*                           4892/dhclient
  udp        0      0 0.0.0.0:111             0.0.0.0:*                           598/rpcbind
  udp        0      0 47.100.XX.XX:323           0.0.0.0:*                           6750/chronyd
  udp        0      0 0.0.0.0:720             0.0.0.0:*                           598/rpcbind
  udp6       0      0 :::111                  :::*                                598/rpcbind
  udp6       0      0 ::1:323                 :::*                                6750/chronyd
  udp6       0      0 fe80::216:XXXX:fe03:546 :::*                                6673/dhclient
  udp6       0      0 :::720                  :::*                                598/rpcbind

  If Proto is tcp, the Pod is listening on an IPv4 address. If it is tcp6, the Pod is listening on an IPv6 address.

• Problem: A Pod can be accessed using its IPv6 address from within the cluster, but not from the internet.

  Cause: The IPv6 address may not have public bandwidth configured.

  Solution: Configure public bandwidth for the IPv6 address. For more information, see Enable and manage IPv6 public bandwidth.

• Problem: The Pod cannot be accessed using the IPv6 cluster IP.

  Solution:

  1. Verify that the spec.ipFamilyPolicy of the Service is not set to SingleStack.

  2. Run the netstat -anp command to verify that the Pod is listening on an IPv6 address.

     Expected output:

     Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
     tcp        0      0 127.0.XX.XX:10248         0.0.0.0:*               LISTEN      8196/kubelet
     tcp        0      0 127.0.XX.XX:41935         0.0.0.0:*               LISTEN      8196/kubelet
     tcp        0      0 0.0.XX.XX:111             0.0.0.0:*               LISTEN      598/rpcbind
     tcp        0      0 0.0.XX.XX:22              0.0.0.0:*               LISTEN      3577/sshd
     tcp6       0      0 :::30500                :::*                    LISTEN      1916680/kube-proxy
     tcp6       0      0 :::10250                :::*                    LISTEN      8196/kubelet
     tcp6       0      0 :::31183                :::*                    LISTEN      1916680/kube-proxy
     tcp6       0      0 :::10255                :::*                    LISTEN      8196/kubelet
     tcp6       0      0 :::111                  :::*                    LISTEN      598/rpcbind
     tcp6       0      0 :::10256                :::*                    LISTEN      1916680/kube-proxy
     tcp6       0      0 :::31641                :::*                    LISTEN      1916680/kube-proxy
     udp        0      0 0.0.0.0:68              0.0.0.0:*                           4892/dhclient
     udp        0      0 0.0.0.0:111             0.0.0.0:*                           598/rpcbind
     udp        0      0 47.100.XX.XX:323           0.0.0.0:*                           6750/chronyd
     udp        0      0 0.0.0.0:720             0.0.0.0:*                           598/rpcbind
     udp6       0      0 :::111                  :::*                                598/rpcbind
     udp6       0      0 ::1:323                 :::*                                6750/chronyd
     udp6       0      0 fe80::216:XXXX:fe03:546 :::*                                6673/dhclient
     udp6       0      0 :::720                  :::*                                598/rpcbind

     If Proto is tcp, the Pod is listening on an IPv4 address. If it is tcp6, the Pod is listening on an IPv6 address.

  3. Problem: The Pod cannot access the internet over IPv6.

     Solution: To enable outbound IPv6 internet access, create an IPv6 Gateway and configure public bandwidth for the IPv6 address. For more information, see Create and manage an IPv6 Gateway and Enable and manage IPv6 public bandwidth.

Insufficient IP addresses in a Terway vSwitch

Problem description

You are unable to create pods. When you log on to the VPC console, select the target region, and view the information for the vSwitch used by the cluster, you will find that its number of available IPs is 0. To confirm the issue, see More information.

Cause

The vSwitch used by Terway on the node has no available IP addresses. This prevents pods from obtaining IP resources, causing them to remain in the ContainerCreating state.

Solution

More information

# Replace [$Node_Name] with the name of the node where your pod is located to find the name of the Terway pod on that node.
kubectl get pod -l app=terway-eniip -n kube-system | grep [$Node_Name] 
# Replace [$Pod_Name] with the name of the Terway pod.
kubectl logs --tail=100 -f [$Pod_Name] -n kube-system -c terway

time="2020-03-17T07:03:40Z" level=warning msg="Assign private ip address failed: Aliyun API Error: RequestId: 2095E971-E473-4BA0-853F-0C41CF52651D Status Code: 403 Code: InvalidVSwitchId.IpNotEnough Message: The specified VSwitch \"vsw-AAA\" has not enough IpAddress., retrying"

Pod IP outside vSwitch CIDR in Terway mode

Symptom

In a Terway network, Pods are assigned IP addresses outside the configured vSwitch CIDR block.

Cause

Terway allocates Pod IP addresses from the VPC and assigns them to containers by using elastic network interfaces (ENIs). You can specify a vSwitch only when creating a new ENI. If an ENI already exists, Pods continue to receive IP addresses from the vSwitch associated with that ENI.

This issue typically occurs in the following two scenarios:

• You add a node previously used in a different cluster that was not drained before removal.

• You manually update the vSwitch configuration used by Terway. Because nodes may still have ENIs associated with the old configuration, new Pods may be assigned IP addresses from these ENIs.

Solution

To apply the new configuration, create new nodes or rotate existing ones.

To rotate an existing node, follow these steps:

1. Drain and remove the node. For more information, see Remove a node.

2. Detach the ENIs from the removed node. For more information, see Manage ENIs.

3. After detaching the ENIs, re-add the node to the original ACK cluster. For more information, see Add existing nodes.

Pod IP assignment failure in Terway network mode

Symptoms

In the Terway network mode, pods cannot obtain IP addresses even after you add a new vSwitch.

Cause

Pod IPs come from VPC addresses and are assigned to containers through elastic network interfaces (ENIs). Terway applies the new vSwitch configuration only when it creates a new ENI. Because the node has reached its ENI quota, Terway cannot create a new ENI. As a result, the new vSwitch configuration does not take effect. For more information about the ENI quota, see Elastic network interfaces overview.

Solution

Create new nodes or rotate existing nodes to apply the new configuration.

To rotate an old node, follow these steps:

1. Drain and remove the old node. For more information, see Remove a node.

2. Detach the ENIs from the removed node. For more information, see Manage ENIs.

3. After the ENIs are detached, add the node back to the original ACK cluster. For more information, see Add an existing node.

Enable in-cluster load balancing in Terway IPvlan

Symptom

For newly created clusters using Terway v1.2.0 or later in IPvlan mode, in-cluster load balancing is enabled by default. When a pod accesses an ExternalIP or a LoadBalancer Service from within the cluster, traffic routes to the Service network. This topic describes how to enable this feature for existing Terway IPvlan clusters.

Cause

Typically, kube-proxy short-circuits traffic from pods to ExternalIP and LoadBalancer Services. Instead of routing traffic externally, kube-proxy redirects requests directly to a backend Endpoint. However, in Terway IPvlan mode, this traffic is handled by Cilium, not kube-proxy. Versions of Terway before v1.2.0 did not support this short-circuiting mechanism. Although the feature is enabled by default in new clusters that use Terway v1.2.0 or later, you must manually enable it for existing clusters.

Solution

1. Run the following command to edit the Terway ConfigMap.

   kubectl edit cm eni-config -n kube-system

2. Add the following line to the eni_conf section.

   in_cluster_loadbalance: "true"

   Note

   Ensure that in_cluster_loadbalance is at the same indentation level as other fields in eni_conf.

3. Run the following command to recreate the Terway pods and apply the configuration.

   kubectl delete pod -n kube-system -l app=terway-eniip

   Verify the configuration

   Run the following command to check the logs of the policy container in a terway-eniip pod. The configuration is successful if the output contains enable-in-cluster-loadbalance=true.

   kubectl logs -n kube-system <terway-pod-name> policy | grep enable-in-cluster-loadbalance

Assign pods to a CIDR block for allowlisting

Symptom

To secure access to services such as databases, you must add client IP addresses to an allowlist. In a container network, this process is challenging because pod IP addresses are dynamic.

Cause

Solution

You can use node labels to specify the vSwitch that pods should use. When a pod is scheduled to a node with a specific label, it receives an IP address from the custom vSwitch defined for that label.

1. Create a ConfigMap named eni-config-fixed in the kube-system namespace to specify the dedicated vSwitch.

   This example uses vsw-2zem796p76viir02c****, which corresponds to the 10.2.1.0/24 CIDR block.

   apiVersion: v1
   data:
     eni_conf: |
       {
          "vswitches": {"cn-beijing-h":["vsw-2zem796p76viir02c****"]},
          "security_group": "sg-bp19k3sj8dk3dcd7****",
          "security_groups": ["sg-bp1b39sjf3v49c33****","sg-bp1bpdfg35tg****"]
       }
   kind: ConfigMap
   metadata:
     name: eni-config-fixed
     namespace: kube-system
   
                           

2. Create a node pool and apply the label terway-config:eni-config-fixed to its nodes. For more information, see Create a node pool.

   To ensure that other pods are not scheduled to nodes in this node pool, you can also apply a taint, such as fixed=true:NoSchedule.

3. Scale out the node pool. For more information, see Scale a node pool manually.

   New nodes in this node pool automatically inherit the label and taint configured in the previous step.

4. Create a deployment that schedules pods to nodes with the terway-config:eni-config-fixed label. The pod specification must include a toleration for the taint.

   apiVersion: apps/v1 # For Kubernetes versions earlier than 1.8.0, use apps/v1beta1.
   kind: Deployment
   metadata:
     name: nginx-fixed
     labels:
       app: nginx-fixed
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: nginx-fixed
     template:
       metadata:
         labels:
           app: nginx-fixed
       spec:
         tolerations:        # Add a toleration to match the taint on the node.
         - key: "fixed"
           operator: "Equal"
           value: "true"
           effect: "NoSchedule"
         nodeSelector:
           terway-config: eni-config-fixed
         containers:
         - name: nginx
           image: nginx:1.9.0 # Replace with your actual image: <image_name:tags>.
           ports:
           - containerPort: 80

   Verification

   1. Run the following command to check the pod IP addresses.

      kubectl get po -o wide | grep fixed

      Expected output:

      nginx-fixed-57d4c9bd97-l****                   1/1     Running             0          39s    10.2.1.124    bj-tw.062149.aliyun.com   <none>           <none>
      nginx-fixed-57d4c9bd97-t****                   1/1     Running             0          39s    10.2.1.125    bj-tw.062148.aliyun.com   <none>           <none>

      The output shows that the pod IP addresses are assigned from the specified vSwitch.

   2. Run the following command to scale out the deployment to 30 pods.

      kubectl scale deployment nginx-fixed --replicas=30

      Expected output:

      nginx-fixed-57d4c9bd97-2****                   1/1     Running     0          60s     10.2.1.132    bj-tw.062148.aliyun.com   <none>           <none>
      nginx-fixed-57d4c9bd97-4****                   1/1     Running     0          60s     10.2.1.144    bj-tw.062149.aliyun.com   <none>           <none>
      nginx-fixed-57d4c9bd97-5****                   1/1     Running     0          60s     10.2.1.143    bj-tw.062148.aliyun.com   <none>           <none>
      ...

      The output shows that all newly created pods are assigned IP addresses from the specified vSwitch. You can now add the CIDR block of this vSwitch to your database's allowlist to grant access to these pods.

Pods cannot ping some ECS instances

Symptom

In Flannel network mode, VPN routes appear normal. However, Pods cannot ping some ECS instances.

Cause

Solution

NodeNetworkUnavailable taint on cluster nodes

Symptom

In a cluster that uses the Flannel network plugin, newly added nodes have the NodeNetworkUnavailable taint, which prevents Pods from being scheduled on them.

Cause

The Cloud Controller Manager fails to promptly remove the taint from the node. This can occur if the route table is full or the VPC contains multiple route tables.

Solution

Run the kubectl describe node command to inspect the node's events and use the resulting error messages to resolve the issue. If your VPC uses multiple route tables, you must manually configure the Cloud Controller Manager to support them. For more details, see Use multiple route tables in a VPC.

Pod startup failure: No IP addresses available

Symptom

In a cluster that uses the Flannel network plugin, Pods fail to start. Inspecting Pod events reveals an error message similar to failed to allocate for range 0: no IP addresses available in range set: 172.30.34.129-172.30.34.190.

Cause

Solution

• To resolve IP address leaks in an ACK cluster running a Kubernetes version earlier than 1.20, upgrade the cluster to version 1.20 or later. For more information, see Manually upgrade a cluster.

• To resolve IP address leaks caused by an older Flannel version, upgrade the Flannel component to v0.15.1.11-7e95fe23-aliyun or later. Follow these steps:

  In Flannel v0.15.1.11-7e95fe23-aliyun and later, ACK moves the IP allocation database to a temporary filesystem (/var/run). This filesystem is automatically cleared on node reboot, which prevents IP address leaks.

  1. Upgrade the Flannel component to v0.15.1.11-7e95fe23-aliyun or later. For more information, see Manage components.

  2. Run the following command to edit the kube-flannel-cfg file, and then add the dataDir and ipam parameters to the kube-flannel-cfg file.

     kubectl -n kube-system edit cm kube-flannel-cfg

     The following code block shows an example of the kube-flannel-cfg file.

     # Before modification
         {
           "name": "cb0",
           "cniVersion":"0.3.1",
           "plugins": [
             {
               "type": "flannel",
               "delegate": {
                 "isDefaultGateway": true,
                 "hairpinMode": true
                },
             },
             # portmap # Optional for earlier versions. Ignore if not in use.
             {
               "type": "portmap",
               "capabilities": {
                 "portMappings": true
               },
               "externalSetMarkChain": "KUBE-MARK-MASQ"
             }
           ]
         }
     
     # After modification
         {
           "name": "cb0",
           "cniVersion":"0.3.1",
           "plugins": [
             {
               "type": "flannel",
               "delegate": {
                 "isDefaultGateway": true,
                 "hairpinMode": true
                },
               # Note the comma.
               "dataDir": "/var/run/cni/flannel",
               "ipam": {
                 "type": "host-local",
                 "dataDir": "/var/run/cni/networks"
                }
             },
             {
               "type": "portmap",
               "capabilities": {
                 "portMappings": true
               },
               "externalSetMarkChain": "KUBE-MARK-MASQ"
             }
           ]
         }

  3. Run the following command to restart the Flannel Pods.

     Restarting the Flannel Pods does not affect running workloads.

     kubectl -n kube-system delete pod -l app=flannel

  4. Delete the IP allocation directories on each node and restart the node.

     1. Drain the node to evict all existing Pods. For more information, see Drain a node and manage its scheduling state.

     2. Log on to the node and run the following commands to delete the IP allocation directories.

        rm -rf /etc/cni/
        rm -rf /var/lib/cni/

     3. Restart the instance. For more information, see Restart an instance.

     4. Repeat these steps on all affected nodes.

  5. Run the following commands on a node to verify that it is using the temporary filesystem.

     if [ -d /var/lib/cni/networks/cb0 ]; then echo "not using tmpfs"; fi
     if [ -d /var/run/cni/networks/cb0 ]; then echo "using tmpfs"; fi
     cat /etc/cni/net.d/10-flannel.conf*

     If the output is using tmpfs, the node is now using the /var/run temporary filesystem for the IP allocation database, and the change was successful.

• If you cannot immediately upgrade ACK or Flannel, you can use the following temporary workaround. This workaround applies to leaks from both causes described above.

  This procedure only cleans up existing leaked IP addresses. The underlying issue may recur until you upgrade the Flannel component or the cluster.

  Note

  • These commands do not apply to nodes running Flannel v0.15.1.11-7e95fe23-aliyun or later and have already switched to using /var/run for IP address allocation.

  • The following scripts are for reference only. They may not work correctly on nodes with custom configurations.

  1. Cordon the affected node to prevent scheduling new Pods on it. For more information, see Drain a node and manage its scheduling state.

  2. Use one of the following scripts to clean up leaked IP addresses on the node, depending on your container runtime.

     • If you use the Docker runtime, use the following script to clean up the node.

       #!/bin/bash
       cd /var/lib/cni/networks/cb0;
       docker ps -q > /tmp/running_container_ids
       find /var/lib/cni/networks/cb0 -regex ".*/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" -printf '%f\n' > /tmp/allocated_ips
       for ip in $(cat /tmp/allocated_ips); do
         cid=$(head -1 $ip | sed 's/\r#g' | cut -c-12)
         grep $cid /tmp/running_container_ids > /dev/null || (echo removing leaked ip $ip && rm $ip)
       done

     • If you use the containerd runtime, use the following script to clean up the node.

       #!/bin/bash
       # install jq
       yum install -y jq
       
       # export all running pod's configs
       crictl -r /run/containerd/containerd.sock pods -s ready -q | xargs -n1 crictl -r /run/containerd/containerd.sock inspectp > /tmp/flannel_ip_gc_all_pods
       
       # export and sort pod ip
       cat /tmp/flannel_ip_gc_all_pods | jq -r '.info.cniResult.Interfaces.eth0.IPConfigs[0].IP' | sort > /tmp/flannel_ip_gc_all_pods_ips
       
       # export flannel's all allocated pod ip
       ls -alh /var/lib/cni/networks/cb0/1* | cut -f7 -d"/" | sort > /tmp/flannel_ip_gc_all_allocated_pod_ips
       
       # print leaked pod ip
       comm -13 /tmp/flannel_ip_gc_all_pods_ips /tmp/flannel_ip_gc_all_allocated_pod_ips > /tmp/flannel_ip_gc_leaked_pod_ip
       
       # clean leaked pod ip
       echo "Found $(cat /tmp/flannel_ip_gc_leaked_pod_ip | wc -l) leaked Pod IP, press <Enter> to clean."
       read sure
       
       # delete leaked pod ip
       for pod_ip in $(cat /tmp/flannel_ip_gc_leaked_pod_ip); do
           rm /var/lib/cni/networks/cb0/${pod_ip}
       done
       
       echo "Leaked Pod IP cleaned, removing temp file."
       rm /tmp/flannel_ip_gc_all_pods_ips /tmp/flannel_ip_gc_all_pods /tmp/flannel_ip_gc_leaked_pod_ip /tmp/flannel_ip_gc_all_allocated_pod_ips

  3. Uncordon the node to make it schedulable again. For more information, see Drain a node and manage its scheduling state.

Immutable network settings

IPs per node, Pod CIDR, and Service CIDR are immutable after cluster creation. Plan your subnets carefully before you create the cluster.

When to configure multiple route tables

In Flannel network mode, you must configure the cloud-controller-manager to support multiple route tables in certain scenarios. For configuration instructions, see Use multiple route tables in a VPC.

Third-party network plugins

No. ACK clusters do not support third-party network plugins. Installing one may disrupt the cluster network.

Pod CIDR exhaustion: the no IP addresses available in range set error

This error occurs in ACK clusters that use the Flannel network plugin. Flannel assigns each node a fixed range of IP addresses from the Pod CIDR. Once a node exhausts its assigned IP addresses, new Pods cannot be created on it. To resolve this, free up some IP addresses or recreate the cluster. For more information on planning your cluster network, see Plan the network for an ACK managed cluster.

Pod capacity in Terway network mode

The pod capacity of a cluster using the Terway network mode depends on the number of IP addresses its underlying ECS instances can support. For more information, see Use the Terway network plugin.

Terway DataPath V2 data plane mode

• Beginning with Terway v1.8.0, new clusters created with the IPvlan option use DataPath V2 mode by default. Existing clusters that use IPvlan continue to use the legacy IPvlan mode.

• DataPath V2 is a next-generation data plane that offers improved compatibility compared to the legacy IPvlan mode. For more information, see Use the Terway network plugin.

Terway network Pod lifecycle

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > DaemonSets.

3. At the top of the DaemonSets page, click and select the kube-system namespace.

4. On the DaemonSets page, search for terway-eniip and click terway-eniip in the Name column.

   The following table describes the Pod statuses.

   Type	Description
   Ready	All containers in the Pod are running.
   Pending	The Pod is waiting for Terway to configure its network resources. The Pod has not been scheduled to a node due to insufficient node resources. For more information, see Troubleshoot Pod exceptions.
   ContainerCreating	The Pod has been scheduled to a node and is waiting for network initialization to complete.

   For more information, see Pod Lifecycle.

Terway upgrade issues

Terway: Pod creation fails with 'MAC address not found' error

 failed to do add; error parse config, can't found dev by mac 00:16:3e:xx:xx:xx: not found

Configure a cluster domain

The default cluster domain for an ACK cluster is cluster.local. You can also specify a custom cluster domain when you create a cluster. Note the following:

• The cluster domain can be configured only during cluster creation and cannot be modified afterward.

• The cluster domain is the top-level domain for in-cluster Service names, creating an isolated DNS zone for internal services. To prevent DNS resolution conflicts, the cluster domain must not overlap with external private or public DNS zones.

  How it works

  ACK uses CoreDNS as its default DNS server. If a custom cluster domain is specified, the default CoreDNS Corefile is configured as follows:

    Corefile: |
      .:53 {
          errors
          log
          health {
             lameduck 15s
          }
          ready
          kubernetes {{.ClusterDomain}} in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
          }
          ...
          forward . /etc/resolv.conf {
            prefer_udp
          }
          ...
        }

  If you do not specify a custom cluster domain, the Corefile configuration is similar to the following:

          kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
          }
          ...
          forward . /etc/resolv.conf {
            prefer_udp
          }

  When CoreDNS handles DNS resolution for the cluster domain, it does not forward these requests to upstream DNS servers by default. This behavior prevents a fallthrough to the forward plugin, ensuring that in-cluster DNS queries are efficient, secure, and isolated from external network issues. If an in-cluster DNS request is mistakenly forwarded to an upstream DNS server, it can trigger a recursive query chain. This long chain can cause the DNS resolution request to exceed its timeout threshold and fail.

  If a custom cluster domain overlaps with an external public top-level domain or a domain defined in Alibaba Cloud DNS PrivateZone, and all Pods in the cluster use CoreDNS as their DNS server, CoreDNS will not forward resolution requests for that domain to an upstream DNS server. Consequently, external domain names within that zone cannot be resolved.

  Furthermore, if the cluster domain overlaps with an external public top-level domain or a domain in an Alibaba Cloud DNS PrivateZone, and a Service name within the cluster also matches a subdomain in that external zone, CoreDNS prioritizes resolving the name to the internal Service IP address instead of the external domain's address. This causes incorrect routing and prevents access to the external service.