Optimize pod network allocation speed for the Terway plugin

Updated at:

Elastic network interfaces (ENIs) cannot be created fast enough for second-level pod startup. Terway caches idle IP addresses in a per-node pool, so pods get an IP without an API call. Most clusters need no configuration changes; to allocate IPs faster, tune the pool size and API rate limits.

For other contributors to pod startup latency, see Customize the kubelet parameters of a node pool. To speed up storage mounting, see Enable disk parallel mounting.

How it works

ENI management model

Terway manages the secondary ENIs of ECS instances and assigns IP addresses to pods. It does not manage a node's primary ENI. As pod count changes, Terway dynamically adjusts both the number of secondary ENIs and the number of IP addresses assigned to each ENI.

image

An ENI is an ECS network resource that can hold multiple IP addresses.

IP address pool

Terway caches available IP addresses in a per-node pool to decouple pod creation from ENI API calls:

  • Pod creation: Terway allocates an IP address from the pool. No API call is needed, which keeps pod startup fast.

  • Pod deletion: The IP address returns to the pool. When idle addresses exceed the configured threshold, Terway calls the ECS API to release the excess back to the ENI.

Pool size directly controls the trade-off between startup speed (more idle IPs held in reserve) and IP address consumption.

CNI execution time

Creating a pod involves kubelet, the container runtime (containerd), and the CNI plugin. After all three complete their work, kubelet reports the pod's IP address and running status to kube-apiserver.

image

Terway network plugin performance metrics

The primary performance metrics of the Terway network plugin are the time required to create and delete pods. Terway is significantly affected by external components, so configuring each component for your workload delivers the best IP allocation performance.

Test scenario: the total time required to create and delete 100 pods on a single node.

The following tables compare the pod creation and deletion results of Terway and Flannel on nodes of the same specification.

Test environment

Item Setting
ECS instance type of the node ecs.ebmg8i.48xlarge
kubelet throttling kubeAPIQPS and kubeAPIBurst are set to 1000
Terway configuration
  • cilium_args = "--api-rate-limit=endpoint-create=rate-limit:200/s,rate-burst:400"

  • min_pool_size = 1000

  • max_pool_size = 1000

  • kube_client_qps = 50

  • kube_client_burst = 100

  • CPU limit of the Terway and Policy containers: 8C

Terway version v1.13.3
Flannel version v0.15.1.23-33d25c1-aliyun

Test results

Network plugin Pod creation Pod deletion
HostNetwork 3s 5s
Flannel 3s 6s
Terway + DataPathV2 13s 13s
Terway 4s 5s
The preceding test results are provided only as a reference for optimization, and are not a commitment of any kind.

Default behavior

By default, Terway caches five idle IP addresses per node and disables IP prefetching. This covers most workloads without any additional configuration, balancing IP density against fast allocation when the cluster scales out.

Tune the IP address pool

Adjust the following parameters in the Terway ConfigMap (eni-config in the kube-system namespace) to match your cluster's requirements. For configuration steps, see Customize the eni_conf parameters in the Terway ConfigMap.

Parameter Type Default Description Configurable in the ACK console
max_pool_size int 5 Maximum number of idle IP addresses retained in the pool per node Yes
min_pool_size int 0 Minimum number of idle IP addresses retained in the pool per node. Cannot exceed max_pool_size. Yes
ip_warm_up_size int — Number of IP addresses to prefetch at a time. This parameter takes effect only for newly added nodes, and the prefetch task stops after the number of IP addresses allocated on the node reaches the expected value. This parameter requires terway-eniip v1.16.4 or later, and the ipam_type of the cluster must be set to crd. No
kube_client_qps float32 — Queries per second (QPS) for the Kubernetes client. Must be configured together with kube_client_burst. No
kube_client_burst int — Burst limit for the Kubernetes client. Must be configured together with kube_client_qps and must be greater than kube_client_qps. No
Important

Decreasing max_pool_size and increasing min_pool_size makes Terway adjust the resource watermark more aggressively until the watermark matches your configuration, which increases the frequency of ECS API calls. If the call rate approaches your quota limit, API throttling may occur. Review Throttling and quota management before making changes, and check the API operations used by Terway in ACK role AliyunCSManagedNetworkRole.

Strategy comparison

Dimension IP address pool One-time prefetch
When it takes effect Continuous. Terway calculates and maintains the number of idle IP addresses based on real-time pod consumption. One-time. A batch of IP addresses is requested only when a node joins the cluster, and the task ends after the batch is allocated.
Resource usage Balanced. IP addresses are reclaimed automatically as workloads scale, and IP addresses beyond max_pool_size are released. The prefetched IP addresses join the IP address pool and follow its release logic.
Primary goal Optimizes pod creation speed in steady state for sustained rapid scale-in and scale-out. Reduces the time required to deploy the first pod on a new node, which addresses the cold-start issue.
Key parameters min_pool_size (lower bound), max_pool_size (upper bound) ip_warm_up_size (number of IP addresses to prefetch)

Reduce IP waste in large clusters

If your cluster uses a small CIDR block and has more than 2,000 nodes, at most 10,000 idle IP addresses can be retained by default. Decrease max_pool_size or configure an idle IP reclaim policy to reduce the number of idle IP addresses.

Idle IP reclaim policy

Assume that the node is configured as follows:

  • min_pool_size = 2

  • max_pool_size = 10

Without an idle IP reclaim policy:

  • The system does not proactively reclaim unused IP addresses on the node.

  • Each node may retain up to 10 idle IP addresses.

With an idle IP reclaim policy:

  • After the reclaim policy is triggered and completes, the system clears the excess idle IP addresses.

  • The number of idle IP addresses on each node is reclaimed to at least min_pool_size. In this example, two idle IP addresses are retained.

  • This mechanism improves IP address utilization and avoids resource waste.

Speed up pod IP allocation

Use one or more of the following approaches based on your requirements:

  • Hold more idle IPs in reserve: Increase max_pool_size. Terway keeps more IP addresses ready on each node, which speeds up pod IP re-allocation. It does not prefetch IP addresses to the node.

  • Enable IP prefetching: Increase min_pool_size above 0. Terway proactively fetches and caches additional IP addresses before demand spikes.

  • Prefetch IP addresses for new nodes: Set ip_warm_up_size to prefetch a batch of IP addresses when a node joins the cluster.

  • Increase Kubernetes API throughput: Raise kubeAPIQPS and kubeAPIBurst in the kubelet configuration. See Customize the kubelet parameters of a node pool.

  • Bypass the CNI plugin entirely: For workloads with strict latency requirements, configure the pod to use host network mode. This skips CNI IP allocation.

Speed up the startup of periodic tasks

  1. If the cluster has sufficient available IP addresses, increase max_pool_size to keep more idle IP addresses on the node.

  2. When available IP addresses are scarce, speeding up the startup of periodic tasks requires a balance between occupying as few idle IP addresses as possible and prefetching precisely before each task window. This topic provides a combination of strategies: prefetch IP addresses on nodes, reuse cached IP addresses within the task window, and reclaim IP addresses promptly after the task ends.

Strategy overview

  • Primary goal: maximize IP address utilization within a limited CIDR block while keeping periodic tasks ready to start quickly in their task window.

  • Principles:

    • Do not permanently raise min_pool_size. A permanently higher min_pool_size forces every node to keep idle IP addresses resident, which tightens the globally allocatable IP pool.

    • Selectively raise max_pool_size on the nodes that run the tasks, to cache IP addresses for fast reuse.

    • Event-driven prefetching: use a temporary pod to trigger IP allocation and keep the IP addresses in the node cache, instead of globally raising min_pool_size.

    • Enable idle IP reclamation: release cached IP addresses as soon as the task completes to avoid long-term occupation.

Parameters and configuration recommendations

  • Watermark parameters

    • max_pool_size: temporarily increase this value only on the nodes that run periodic tasks. Estimate the number of IP addresses each node needs from the number of new pods and the number of nodes.

    • min_pool_size: keep it at 0. Trigger prefetching with a temporary pod.

  • Idle IP reclaim policy

    • idle_ip_reclaim_after: set it to match the task duration and the cooldown time. For example, if a periodic task runs for 10 to 20 minutes, configure "20m"~"45m".

    • idle_ip_reclaim_interval: the interval between reclaim batches, for example "5m"~"10m".

    • idle_ip_reclaim_batch_size: the maximum number of IP addresses to reclaim in each batch. Recommended value: 5.

    • idle_ip_reclaim_jitter_factor: "0.1"~"0.3". Prevents transient spikes caused by simultaneous reclamation. A larger value disperses reclamation further.

Resolve endpoint-creation throttling in Datapath V2 or IPvlan mode

In Datapath V2 or IPvlan mode, when many pods start in parallel on a single node, Terway's agent rate-limits endpoint creation. The default limit is 0.5 per second. When the limit is reached, the following event appears:

unable to create endpoint: [PUT /endpoint/{id}][429]

The steps below raise this limit by setting the cilium_args field in the eni-config ConfigMap.

Prerequisites

Before you begin, ensure that you have:

Warning
  • Editing Terway configuration with kubectl is a high-risk operation. kubectl does not validate input. Incorrect values can suspend Terway. Verify all parameters before saving.

  • Raising the endpoint-creation rate limit increases Terway's CPU and memory usage and may increase the load on kube-apiserver.

Raise the endpoint-creation rate limit

  1. Open the eni-config ConfigMap for editing:

    kubectl edit cm -n kube-system eni-config
  2. In the 10-terway.conf section, add the cilium_args field. The following example sets the creation rate to 2 per second with a burst limit of 5:

    10-terway.conf: |
        {
            "cniVersion": "0.4.0",
            "name": "terway",
            "capabilities": {"bandwidth": true},
            "eniip_virtual_type": "IPvlan",
            "cilium_args": "--api-rate-limit=endpoint-create=rate-limit:2/s,rate-burst:5",
            "type": "terway"
        }
  3. Save and close the file, then restart the Terway DaemonSet to apply the change:

    kubectl rollout restart -n kube-system daemonset.apps/terway-eniip

What's next