Optimize pod network allocation speed for the Terway plugin
Elastic network interfaces (ENIs) cannot be created fast enough for second-level pod startup. Terway caches idle IP addresses in a per-node pool, so pods get an IP without an API call. Most clusters need no configuration changes; to allocate IPs faster, tune the pool size and API rate limits.
For other contributors to pod startup latency, see Customize the kubelet parameters of a node pool. To speed up storage mounting, see Enable disk parallel mounting.
How it works
ENI management model
Terway manages the secondary ENIs of ECS instances and assigns IP addresses to pods. It does not manage a node's primary ENI. As pod count changes, Terway dynamically adjusts both the number of secondary ENIs and the number of IP addresses assigned to each ENI.
An ENI is an ECS network resource that can hold multiple IP addresses.
IP address pool
Terway caches available IP addresses in a per-node pool to decouple pod creation from ENI API calls:
-
Pod creation: Terway allocates an IP address from the pool. No API call is needed, which keeps pod startup fast.
-
Pod deletion: The IP address returns to the pool. When idle addresses exceed the configured threshold, Terway calls the ECS API to release the excess back to the ENI.
Pool size directly controls the trade-off between startup speed (more idle IPs held in reserve) and IP address consumption.
CNI execution time
Creating a pod involves kubelet, the container runtime (containerd), and the CNI plugin. After all three complete their work, kubelet reports the pod's IP address and running status to kube-apiserver.
Terway network plugin performance metrics
The primary performance metrics of the Terway network plugin are the time required to create and delete pods. Terway is significantly affected by external components, so configuring each component for your workload delivers the best IP allocation performance.
Test scenario: the total time required to create and delete 100 pods on a single node.
The following tables compare the pod creation and deletion results of Terway and Flannel on nodes of the same specification.
Test environment
| Item | Setting |
|---|---|
| ECS instance type of the node | ecs.ebmg8i.48xlarge |
| kubelet throttling | kubeAPIQPS and kubeAPIBurst are set to 1000 |
| Terway configuration |
|
| Terway version | v1.13.3 |
| Flannel version | v0.15.1.23-33d25c1-aliyun |
Test results
| Network plugin | Pod creation | Pod deletion |
|---|---|---|
| HostNetwork | 3s | 5s |
| Flannel | 3s | 6s |
| Terway + DataPathV2 | 13s | 13s |
| Terway | 4s | 5s |
The preceding test results are provided only as a reference for optimization, and are not a commitment of any kind.
Default behavior
By default, Terway caches five idle IP addresses per node and disables IP prefetching. This covers most workloads without any additional configuration, balancing IP density against fast allocation when the cluster scales out.
Tune the IP address pool
Adjust the following parameters in the Terway ConfigMap (eni-config in the kube-system namespace) to match your cluster's requirements. For configuration steps, see Customize the eni_conf parameters in the Terway ConfigMap.
| Parameter | Type | Default | Description | Configurable in the ACK console |
|---|---|---|---|---|
max_pool_size |
int | 5 |
Maximum number of idle IP addresses retained in the pool per node | Yes |
min_pool_size |
int | 0 |
Minimum number of idle IP addresses retained in the pool per node. Cannot exceed max_pool_size. |
Yes |
ip_warm_up_size |
int | — | Number of IP addresses to prefetch at a time. This parameter takes effect only for newly added nodes, and the prefetch task stops after the number of IP addresses allocated on the node reaches the expected value. This parameter requires terway-eniip v1.16.4 or later, and the ipam_type of the cluster must be set to crd. |
No |
kube_client_qps |
float32 | — | Queries per second (QPS) for the Kubernetes client. Must be configured together with kube_client_burst. |
No |
kube_client_burst |
int | — | Burst limit for the Kubernetes client. Must be configured together with kube_client_qps and must be greater than kube_client_qps. |
No |
Decreasing max_pool_size and increasing min_pool_size makes Terway adjust the resource watermark more aggressively until the watermark matches your configuration, which increases the frequency of ECS API calls. If the call rate approaches your quota limit, API throttling may occur. Review Throttling and quota management before making changes, and check the API operations used by Terway in ACK role AliyunCSManagedNetworkRole.
Strategy comparison
| Dimension | IP address pool | One-time prefetch |
|---|---|---|
| When it takes effect | Continuous. Terway calculates and maintains the number of idle IP addresses based on real-time pod consumption. | One-time. A batch of IP addresses is requested only when a node joins the cluster, and the task ends after the batch is allocated. |
| Resource usage | Balanced. IP addresses are reclaimed automatically as workloads scale, and IP addresses beyond max_pool_size are released. |
The prefetched IP addresses join the IP address pool and follow its release logic. |
| Primary goal | Optimizes pod creation speed in steady state for sustained rapid scale-in and scale-out. | Reduces the time required to deploy the first pod on a new node, which addresses the cold-start issue. |
| Key parameters | min_pool_size (lower bound), max_pool_size (upper bound) |
ip_warm_up_size (number of IP addresses to prefetch) |
Reduce IP waste in large clusters
If your cluster uses a small CIDR block and has more than 2,000 nodes, at most 10,000 idle IP addresses can be retained by default. Decrease max_pool_size or configure an idle IP reclaim policy to reduce the number of idle IP addresses.
Idle IP reclaim policy
Assume that the node is configured as follows:
-
min_pool_size = 2 -
max_pool_size = 10
Without an idle IP reclaim policy:
-
The system does not proactively reclaim unused IP addresses on the node.
-
Each node may retain up to 10 idle IP addresses.
With an idle IP reclaim policy:
-
After the reclaim policy is triggered and completes, the system clears the excess idle IP addresses.
-
The number of idle IP addresses on each node is reclaimed to at least
min_pool_size. In this example, two idle IP addresses are retained. -
This mechanism improves IP address utilization and avoids resource waste.
Speed up pod IP allocation
Use one or more of the following approaches based on your requirements:
-
Hold more idle IPs in reserve: Increase
max_pool_size. Terway keeps more IP addresses ready on each node, which speeds up pod IP re-allocation. It does not prefetch IP addresses to the node. -
Enable IP prefetching: Increase
min_pool_sizeabove0. Terway proactively fetches and caches additional IP addresses before demand spikes. -
Prefetch IP addresses for new nodes: Set
ip_warm_up_sizeto prefetch a batch of IP addresses when a node joins the cluster. -
Increase Kubernetes API throughput: Raise
kubeAPIQPSandkubeAPIBurstin the kubelet configuration. See Customize the kubelet parameters of a node pool. -
Bypass the CNI plugin entirely: For workloads with strict latency requirements, configure the pod to use
host networkmode. This skips CNI IP allocation.
Speed up the startup of periodic tasks
-
If the cluster has sufficient available IP addresses, increase
max_pool_sizeto keep more idle IP addresses on the node. -
When available IP addresses are scarce, speeding up the startup of periodic tasks requires a balance between occupying as few idle IP addresses as possible and prefetching precisely before each task window. This topic provides a combination of strategies: prefetch IP addresses on nodes, reuse cached IP addresses within the task window, and reclaim IP addresses promptly after the task ends.
Strategy overview
-
Primary goal: maximize IP address utilization within a limited CIDR block while keeping periodic tasks ready to start quickly in their task window.
-
Principles:
-
Do not permanently raise
min_pool_size. A permanently highermin_pool_sizeforces every node to keep idle IP addresses resident, which tightens the globally allocatable IP pool. -
Selectively raise
max_pool_sizeon the nodes that run the tasks, to cache IP addresses for fast reuse. -
Event-driven prefetching: use a temporary pod to trigger IP allocation and keep the IP addresses in the node cache, instead of globally raising
min_pool_size. -
Enable idle IP reclamation: release cached IP addresses as soon as the task completes to avoid long-term occupation.
-
Parameters and configuration recommendations
-
Watermark parameters
-
max_pool_size: temporarily increase this value only on the nodes that run periodic tasks. Estimate the number of IP addresses each node needs from the number of new pods and the number of nodes. -
min_pool_size: keep it at0. Trigger prefetching with a temporary pod.
-
-
Idle IP reclaim policy
-
idle_ip_reclaim_after: set it to match the task duration and the cooldown time. For example, if a periodic task runs for 10 to 20 minutes, configure"20m"~"45m". -
idle_ip_reclaim_interval: the interval between reclaim batches, for example"5m"~"10m". -
idle_ip_reclaim_batch_size: the maximum number of IP addresses to reclaim in each batch. Recommended value:5. -
idle_ip_reclaim_jitter_factor:"0.1"~"0.3". Prevents transient spikes caused by simultaneous reclamation. A larger value disperses reclamation further.
-
Resolve endpoint-creation throttling in Datapath V2 or IPvlan mode
In Datapath V2 or IPvlan mode, when many pods start in parallel on a single node, Terway's agent rate-limits endpoint creation. The default limit is 0.5 per second. When the limit is reached, the following event appears:
unable to create endpoint: [PUT /endpoint/{id}][429]
The steps below raise this limit by setting the cilium_args field in the eni-config ConfigMap.
Prerequisites
Before you begin, ensure that you have:
-
kubectl configured to connect to the target ACK cluster. See Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster
-
Editing Terway configuration with kubectl is a high-risk operation. kubectl does not validate input. Incorrect values can suspend Terway. Verify all parameters before saving.
-
Raising the endpoint-creation rate limit increases Terway's CPU and memory usage and may increase the load on kube-apiserver.
Raise the endpoint-creation rate limit
-
Open the
eni-configConfigMap for editing:kubectl edit cm -n kube-system eni-config -
In the
10-terway.confsection, add thecilium_argsfield. The following example sets the creation rate to 2 per second with a burst limit of 5:10-terway.conf: | { "cniVersion": "0.4.0", "name": "terway", "capabilities": {"bandwidth": true}, "eniip_virtual_type": "IPvlan", "cilium_args": "--api-rate-limit=endpoint-create=rate-limit:2/s,rate-burst:5", "type": "terway" } -
Save and close the file, then restart the Terway DaemonSet to apply the change:
kubectl rollout restart -n kube-system daemonset.apps/terway-eniip