Terway network plugin

Before you begin

To better understand the working mode of Terway, read this topic before using the Terway network plug-in.

Make sure you understand the terms used for container network plug-ins and have selected one. For more information, see Networking overview and Comparison between Terway and Flannel.

Before creating a cluster, you must plan its CIDR blocks. For more information, see Plan CIDR blocks for an ACK managed cluster.

Billing

The Terway plugin is free of charge. However, the pods it uses are deployed on each node and consume a small amount of node resources. For details on billing for ACK cloud services, see Cloud resource fees.

Important notes

The Terway configuration file eni-config contains many system parameters. Modifying or deleting unsupported fields can cause network interruptions and pod creation failures. For a list of modifiable parameters, see Customize Terway configuration parameters.

The Terway component uses CRDs to track resource status. Incorrectly modifying system resources can cause network interruptions and pod creation failures.

Pod limit calculation

With the Terway network plugin, a node's maximum pod count depends on the number of elastic network interfaces (ENIs) that its ECS instance type supports. Terway requires nodes to meet a minimum pod limit before they can be added to a cluster. The following table details these calculations.

Install Terway when you create a cluster

You must install the Terway network plug-in when you create a cluster. You cannot change the network plug-in for an existing cluster.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click Create Kubernetes Cluster.

3. Configure the key network parameters for Terway. For more information about other cluster parameters, see Create an ACK managed cluster.

   Parameter	Description
   IPv6 Dual-stack	Select Enable to create a dual-stack cluster that supports both IPv4 and IPv6 addresses. Supported only for Kubernetes 1.22 or later, only with Terway, and cannot be used together with eRDMA. The cluster supports both IPv4 and IPv6 protocols, but communication between worker nodes and the control plane still uses IPv4 addresses. Ensure the following: The cluster VPC supports IPv6 dual-stack. When using Terway in shared ENI mode, the instance type of the node must support IPv6 and have the same number of assignable IPv4 and IPv6 addresses.
   VPC	The VPC for the cluster.
   Network Plug-in	Select Terway.
   DataPath V2	Select this option to use the DataPath V2 acceleration mode. This mode uses a different traffic forwarding path than the standard shared ENI mode, delivering faster network connectivity. For more information, see Network acceleration. Note For new clusters of Kubernetes version 1.34 or later with DataPath V2 enabled, kube-proxy no longer runs on Terway nodes. This mode provides built-in support for portmap, so you do not need to configure the portmap plug-in. For more information, see Configure a custom CNI chain. DataPath V2 is supported only on the following operating system images and requires Linux kernel version 5.10 or later: Alibaba Cloud Linux 3 (all versions) ContainerOS Ubuntu When this feature is enabled, the Terway policy container consumes an additional 0.5 CPU cores and 512 MB of memory on each worker node. This overhead increases as the cluster size grows. In the default Terway configuration, the policy container has a CPU limit of 1 core and no memory limit. In DataPath V2 mode, container network connection tracking (conntrack) information is stored in an eBPF map. Similar to the native conntrack mechanism in Linux, eBPF conntrack uses a Least Recently Used (LRU) algorithm. When the map capacity is reached, the oldest connection records are automatically evicted to accommodate new connections. Configure the relevant parameters based on your workload to prevent the number of connections from exceeding the limit. For more information, see Optimize conntrack configurations in Terway.
   NetworkPolicy support	Enables support for native Kubernetes NetworkPolicy. Note Starting from Terway v1.9.2, NetworkPolicy for new clusters is implemented by eBPF, and DataPath V2 is enabled for the data plane. The console-based management feature for NetworkPolicy is in public preview. To use this feature, submit a request in Quota Center console.
   Support for ENI Trunking	Enables Trunk ENI mode. In this mode, you can configure a static IP address, a dedicated vSwitch, and a security group for each pod. Note You can enable the Trunk ENI option for an ACK managed cluster without submitting a request. If you want to enable Trunk ENI in an ACK dedicated cluster, you must first submit a request in Quota Center console. Starting from Kubernetes 1.31, the Trunk ENI feature is automatically enabled for new ACK managed clusters. You do not need to manually select this option.
   vSwitch	The vSwitches for cluster nodes. We recommend selecting vSwitches from at least three different zones for higher cluster availability.
   Pod vSwitch	The vSwitches for pods. These CIDR blocks can overlap with the node vSwitch CIDR blocks.
   Service CIDR	The CIDR block for Services, which cannot overlap with the node or pod CIDR blocks.
   IPv6 Service CIDR Block	You can configure this parameter after you enable IPv6 dual-stack.

Terway network modes

The following sections compare and explain the different Terway network modes.

Shared ENI mode and exclusive ENI mode

When Terway assigns an IP address to a pod, it operates in one of two modes: shared ENI mode or exclusive ENI mode.

Important

• In Terway v1.11.0 and later, you can select either shared ENI mode or exclusive ENI mode for a node pool within a single cluster. This option is no longer available during cluster creation.

• The primary elastic network interface (ENI) on a node is assigned to the node's operating system. The remaining ENIs are managed by Terway to configure pod networks. Do not manually configure these Terway-managed ENIs. If you need to manage some ENIs yourself, see Configure a filter for ENIs.

Item		Shared ENI mode	Exclusive ENI mode
Pod IP address management	ENI allocation	Multiple pods share an ENI.	Each pod is assigned a separate ENI on its node.
	Pod deployment density	High. A single node can support hundreds of pods.	Low. A typical node can support only a single-digit number of pods.
	Network architecture
	Data link	Pod-to-pod and Service backend traffic passes through the node network protocol stack.	Traffic to Services passes through the node network protocol stack. However, pod-to-pod and Service backend traffic bypasses the stack by using the attached ENI, which results in higher performance.
Use cases		Standard Kubernetes use cases.	This mode delivers network performance close to that of traditional virtual machines. It is suitable for workloads that require high network throughput or low latency.
Network acceleration		Supports DataPath V2 network acceleration. For more information, see Network acceleration.	Network acceleration is not supported. However, pods have dedicated ENIs, which ensures excellent network performance.
NetworkPolicy support		Supports native Kubernetes NetworkPolicy to provide policy-based access control. For more information, see NetworkPolicy support.	NetworkPolicy is not supported.
Node-level network configuration		Supported. See Node-level network configuration.	Supported. See Node-level network configuration.
Access control		When Trunk ENI is enabled, you can configure a static IP address, a separate vSwitch, and a separate security group for each pod. For more information, see Configure a static IP address, separate vSwitch, and separate security group for a pod.	You can configure a static IP address, a separate vSwitch, and a separate security group for each pod.

Network acceleration

In shared ENI mode, you can enable network acceleration to improve performance. This feature uses a different traffic forwarding path from the standard mode. Terway currently supports the DataPath V2 acceleration mode, which is detailed below.

Important

• DataPath V2 is an upgraded version of the IPVLAN + eBPF acceleration mode. For clusters that use Terway v1.8.0 or later, DataPath V2 is the only available acceleration option.

• The DataPath V2 and IPVLAN + eBPF acceleration modes apply only to shared ENI node pools and do not affect exclusive ENI node pools.

Features	Description
Applicable Terway version	Clusters created with Terway v1.8.0 or later.
Network architecture
Accelerated data link	When a pod accesses a Service, eBPF resolves the Service IP address to the IP address of a backend pod. When a pod accesses a pod on a different node, eBPF bypasses the network protocol stacks of both nodes. When a pod accesses another pod on the same node, traffic bypasses the node network protocol stack and is forwarded directly within the node.
Performance optimization	The forwarding process for pod networking on the host is simplified, making pod network performance nearly identical to that of the host. This reduces latency by 30% compared to the regular mode. Service networking uses eBPF to replace the original kube-proxy mode, bypassing iptables or IPVS forwarding on the node. This significantly reduces request latency and provides better scalability with less performance impact in large-scale clusters. Pod NetworkPolicy is also implemented with eBPF, replacing the original iptables implementation. This avoids generating a large number of iptables rules on the host and reduces the impact of network policies on network performance.
How to use	When you create a cluster, select Terway as the Network Plug-in and select the DataPath V2 option.
Usage notes	Requires kernel version 5.10 or later. We recommend that you use Alibaba Cloud Linux OS images. The Sandboxed-Container runtime is not supported. Limitations on NetworkPolicy: The CIDR selector does not support control over pod CIDR blocks. To control access to pods, you must use pod label selectors. The except keyword in CIDR selectors is not fully supported. We recommend that you do not use the except keyword. Using an Egress NetworkPolicy causes access to pods that use the host network and to node IP addresses in the cluster to fail. When you access an externally exposed SLB instance for a LoadBalancer Service from within the cluster, loopback issues may cause network failures. For more information, see Why am I unable to access a load balancer? IPv6 hairpin access is not supported. Limitations on NodePort: If you access a service with ExternalTrafficPolicy=Local, connections may fail. To resolve this issue, change the setting to ExternalTrafficPolicy=Cluster. When you use ExternalTrafficPolicy=Cluster, source network address translation (SNAT) is required for the source address. The available port range is 32768 to 65535.

The following section describes the IPVLAN + eBPF acceleration mode, which was available for older clusters.

IPVLAN + eBPF acceleration mode

Features	Description
Applicable Terway version	Clusters created with Terway v1.7.0 or earlier.
Network architecture
Accelerated data link	When a pod accesses a Service, eBPF resolves the Service IP address to the IP address of a backend pod within the pod's network namespace. When a pod accesses another pod, IPVLAN bypasses the network protocol stacks of both nodes.
How to use	When you create a cluster, select Terway as the Network Plug-in and select the Pod IPVLAN option.

Access control

The shared ENI mode provides more granular control over cluster network traffic through its support for NetworkPolicy and the Trunk ENI option. The exclusive ENI mode also offers some traffic control capabilities.

NetworkPolicy

• Terway exclusive ENI node pools do not support NetworkPolicy.

• Terway shared ENI node pools support the native Kubernetes NetworkPolicy feature, which controls network traffic between pods by using user-defined rules.

  To enable NetworkPolicy in a cluster, select Terway as the Network Plug-in and select the NetworkPolicy Support option when you create the cluster. For more information, see Use NetworkPolicies in ACK clusters.

  Note

  The console-based management feature for NetworkPolicy is in public preview. To use this feature, submit a request in Quota Center console.

Static IP and resources

• By default, Terway exclusive ENI node pools allow you to configure each pod with a static IP address, a separate vSwitch, and a separate security group. This provides fine-grained traffic management and isolation, policy configuration, and IP address management.

• Trunk ENI is an option for Terway shared ENI node pools. When Trunk ENI is enabled, you can configure each pod with a static IP address, a separate vSwitch, and a separate security group.

  To enable this feature, select Terway as the Network Plug-in and select the Support for ENI Trunking option when you create a cluster. For more information, see Configure a static IP address, separate vSwitch, and separate security group for a pod.

  Note

  • You can enable the Trunk ENI option for an ACK managed cluster without submitting a request. If you want to enable Trunk ENI in an ACK dedicated cluster, you must first submit a request in Quota Center console.

  • Starting from Kubernetes 1.31, the Trunk ENI feature is automatically enabled for new ACK managed clusters. You do not need to manually select this option.

  • After you enable Trunk ENI mode, the system installs the terway-eniip and terway-controlplane components.

Scaling limits

Terway manages node network interfaces and IP addresses using OpenAPI calls to cloud products. For OpenAPI usage limits, refer to the documentation of the corresponding cloud product.

• shared ENI mode: Maximum concurrently allocated nodes: 500.

• exclusive ENI/TrunkENI mode: Maximum concurrently allocated Pods: 100.

These quotas are fixed.

Data plane configuration

Terway's networking functionality depends on the precise order and integrity of its kernel-level configuration. Uncoordinated modifications to IP Rule, IP Route, and eBPF hooks by external components (including priority adjustments, rule overrides, or program unloading) can cause critical failures such as pod network interruptions, policy failures, and traffic hijacking. All third-party components must be strictly validated before integration to prevent conflicts.

TC Filter rules

IP rules

FAQ