Best practices for Datapath V2
After you enable Datapath V2 in a cluster that uses the Terway network plugin, optimize conntrack parameters, identity resources, and BPF map capacity to improve cluster performance and stability.
Optimize conntrack configurations
Conntrack is the connection tracking module in the Linux kernel and is used to track the state of network connections. In Datapath V2 mode, conntrack for the container network is provided by eBPF programs. Refer to Optimize conntrack configurations in Terway to adjust conntrack parameters, such as the conntrack table size and the timeout period of TCP connection entries. Increase these values to meet the network requirements of high-concurrency services.
Limit the number of identities
In Datapath V2, NetworkPolicy is implemented through eBPF. Unlike the traditional Netfilter-based approach, the eBPF implementation assigns an identity to each pod to serve as its identity ID, which enables precise management of network permissions.
An identity is derived from a combination of pod labels and namespace labels. Pods that share the same label combination are assigned the same identity so that their network access rules can be managed uniformly.
NetworkPolicy rules use IP-to-identity mappings to determine whether traffic complies with a policy. In simple terms, the system uses the IP address of a pod and its identity to determine whether to allow the traffic.
-
If you do not enable NetworkPolicy, identities serve no purpose, and Terway automatically limits the number of identities.
-
If you enable NetworkPolicy, you should avoid having pods that share the same identity carry different labels. Using different labels generates a large number of identity resources, which can increase the management load on the cluster and slow down IP address allocation.
Configuring label filtering prevents invalid labels from being recorded in identities.
-
Adjusting label filtering rules triggers the creation of new identities. The number of identities increases over a short period, and the resource overhead of API Server also increases to varying degrees.
-
Incorrectly configured label filtering rules cause NetworkPolicy rules to fail.
-
Do not filter out all labels: at least one label (a namespace label or a pod label) must be retained for each group of pods. Otherwise, the system cannot identify the identity of the pod.
-
Filtered labels cannot be used in policy rules: if a label is added to the filter list, the label can no longer be referenced in NetworkPolicy rules. Make sure that you filter only invalid labels.
-
For information about the syntax used to configure label filtering, see Cilium.
To configure label filtering in Terway, see Customize Terway configuration parameters for details about cilium_args.
The number of identities directly affects the number of policy map entries for each pod. The policy map capacity is controlled by bpf-policy-map-max (default: 16384; maximum per endpoint: 65536). For more information, see bpf-policy-map-max.
Monitor BPF map utilization
Once a BPF map is full, pods experience network anomalies or Service entries fail to be programmed. Therefore, enable Prometheus metrics collection for the Cilium Agent in production clusters and configure utilization alerts for all types of BPF maps.
Enable Prometheus metrics
Datapath V2 reuses the metrics endpoint of the Cilium Agent. In the Terway ConfigMap, extend the cilium_args field by appending --prometheus-serve-addr:
"cilium_args": "--prometheus-serve-addr=:9962"
-
:9962is the metrics port agreed upon by the Cilium upstream project. Set it to an empty string to disable metrics. It listens on0.0.0.0, and metrics can be scraped directly through the hostNetwork port of the pod. -
Restart the terway-eniip pod for the change to take effect.
-
Add scrape annotations to the pod or Service, or create a ServiceMonitor, so that Prometheus in the cluster can automatically discover the metrics endpoint:
metadata: annotations: prometheus.io/scrape: "true" prometheus.io/port: "9962" prometheus.io/path: "/metrics"
Configure appropriate security groups for the cluster to prevent external access to the metrics port.
Key BPF map metrics
Some metrics are available only in Terway v1.14.0 or later. Upgrade to the latest version.
|
Metric name |
Type |
Description |
|
|
Gauge |
Most important. The ratio of the map's current entries to |
|
|
Gauge |
Map capacity ( |
|
|
Counter |
Map operation counts, broken down by the |
|
|
Gauge |
The upper limit of the virtual memory occupied by all maps, used to monitor node memory pressure. |
Recommended alerts and query examples
# Any BPF map utilization exceeds 80% for 5 minutes
max by (map_name) (cilium_bpf_map_pressure) > 0.8
# LB Service map utilization per node
cilium_bpf_map_pressure{map_name=~"cilium_lb[46]_services_v2"}
# Policy map utilization per endpoint (top 10)
topk(10, cilium_bpf_map_pressure{map_name=~"cilium_policy_.*"})
# Node Map capacity alert (if >80%, increase bpf-node-map-max)
cilium_bpf_map_pressure{map_name="cilium_node_map_v2"} > 0.8
# Global conntrack map pressure (linked with bpf-map-dynamic-size-ratio tuning)
cilium_bpf_map_pressure{map_name=~"cilium_ct.*_global"} > 0.7
# Map write failure rate
sum by (map_name) (rate(cilium_bpf_map_ops_total{operation="update",outcome="fail"}[5m])) > 0
If you observe that cilium_bpf_map_pressure stays above 0.8, consult the BPF map capacity tuning section and adjust the corresponding parameters by map category. If it exceeds 0.95, treat it as an emergency and expand the capacity immediately; otherwise, entry programming failures may occur within a short time.
BPF map capacity tuning
Datapath V2 relies on many BPF maps to work. Once a map is created, its max_entries is fixed and cannot be expanded at runtime. If the map is filled up, issues occur such as update failures, abnormal pod traffic, and identity/Service entries that cannot be programmed. Use the metrics in the Monitor BPF map utilization section to evaluate map sizes and adjust them in advance.
Adjusting the size of any BPF map requires restarting the Cilium Agent on the Terway node (that is, restarting the terway-eniip pod) to rebuild the map.
During the rebuild, the conntrack/Service entries of existing connections need to be repopulated, which may cause brief connection jitter. Avoid business peak hours and roll out the change in batches.
Parameter overview
|
Parameter |
Function |
Default value |
Value range |
When to adjust |
|
|
Connection tracking (CT) and NAT: the number of TCP/UDP connections a node can keep alive at the same time, and the number of NodePort / source NAT (SNAT) translation entries. When the map is full, new connections fail and packets are dropped |
|
|
The node runs high-concurrency or long-lived-connection workloads, and CT/NAT map pressure is high |
|
|
NetworkPolicy (per pod): the number of policy rules that the data plane of a single pod can hold (identity × port/protocol combinations that are allowed to communicate). When the map is full, policy rule programming is truncated and the actual behavior does not match expectations |
|
|
NetworkPolicy is enabled and there are many identities, or a single pod accesses a wide range of peers |
|
|
Kubernetes Service (all nodes): the total number of entries programmed in the data plane for ClusterIP / NodePort / LoadBalancer. When the map is full, new Services or backend pods cannot be programmed, and Service IP addresses are unreachable bpf-lb-map-max controls the entry capacity of the load balancing (LB) map, not the number of Service IDs. Each Service frontend typically requires one Service ID and occupies multiple Service map entries based on the number of backends. Expanding the LB map can resolve insufficient entry capacity but cannot overcome the 16-bit Service ID space limit. |
|
No enforced upper limit (subject to memory) |
The number of Services in the cluster × the average number of backend pods approaches the default of 65536 |
|
|
Cluster node scale: the number of recognizable Node ID-to-IP mappings; cross-node pod communication, IPsec/WireGuard encryption, and Egress Gateway all depend on this map. When the map is full, newly added nodes cannot participate in the data plane |
|
|
The number of nodes in the cluster approaches 16,000 |
These parameters are configured centrally through the cilium_args field of the Terway ConfigMap. For example:
"cilium_args": "--bpf-map-dynamic-size-ratio=0.005 --bpf-policy-map-max=32768 --bpf-lb-map-max=131072"
To configure these parameters in Terway, see Customize Terway configuration parameters for details about cilium_args.
bpf-map-dynamic-size-ratio
-
Function: on the node, determines the number of connections that can be tracked simultaneously and the number of NodePort/SNAT translation entries. The capacity of the map = the physical memory of the node × this ratio / the size of a single entry. When the capacity is full, new connections cannot be established, and existing connections may also be evicted, causing packet loss.
-
Scope: CT (
cilium_ct4_global/cilium_ct_any4_global), NAT (cilium_snat_v4_external), Neighbor (cilium_nodeport_neigh4), and SockRevNAT. The capacities of these global maps are proportional to this parameter × the physical memory of the node and are dynamically calculated. -
Value range:
(0.0, 1.0], with a default of0.0025(that is, about 0.25% of the physical memory). -
Typical calculation (based on the default element sizes in Cilium):
Node physical memory
CT TCP entries
CT Any entries
NAT entries
512 MiB
33,140
16,570
33,140
1 GiB
66,280
33,140
66,280
4 GiB
265,121
132,560
265,121
16 GiB
1,060,485
530,242
1,060,485
-
Bounds protection: The dynamically calculated result is clamped to
[1Ki, 16Mi](approximately 1 GiB of memory per map), and the per-map ceiling is2^24 = 16,777,216entries. -
Tuning recommendations:
-
For nodes that run high-concurrency services (more than 10,000 long-lived connections per node), increase this ratio moderately to
0.005–0.01. This allows CT/NAT to hold more entries on nodes with large memory. -
Maps with an explicit
bpf-ct-global-tcp-max/bpf-ct-global-any-max/bpf-nat-global-maxsetting are not overridden by this parameter. -
Use
Monitor BPF map utilizationto check map pressure. If maps such ascilium_ct4_globalreport acilium_bpf_map_pressureabove 0.8, adjust this ratio first or explicitly specify a capacity for the corresponding maps.
-
bpf-policy-map-max
-
Scope: Each endpoint (each pod) holds its own
cilium_policy_*map that stores the peer identity × port/protocol entries that allow the endpoint to send and receive traffic. The map is used for NetworkPolicy enforcement in the data plane. -
Default value:
16384. Minimum value:256(2^8). Maximum value:65536(2^16). Values outside this range are automatically clipped, and a warning is printed. -
When to adjust:
-
NetworkPolicy is enabled in the cluster and the number of identities is large (see the
Limit the number of identitiessection). -
The number of peer identities that a single pod needs to access × the number of exposed port/protocol combinations approaches the default value of 16384.
-
The metric
cilium_bpf_map_pressure{map_name=~"cilium_policy_.*"}shows that the pressure remains >0.8.
-
-
How to adjust: In the
cilium_argsfield of the Terway ConfigMap, append--bpf-policy-map-max=32768, and restart the terway-eniip pods for the change to take effect.
The memory occupied by each map is proportional to max_entries, and the number of maps on each node equals the number of pods on that node. Blindly setting the value to 65536 significantly increases the memory overhead of the node. Evaluate this setting based on the actual number of identities.
bpf-node-map-max
-
Function: determines the maximum number of nodes that the cluster can recognize. A single
cilium_node_map_v2map stores the Node ID-to-IP mappings of all nodes in the cluster. The map underpins node-level features such as cross-node pod communication, IPsec/WireGuard encryption, and Egress Gateway. When the map is full, newly added nodes cannot be recognized by the data plane, and cross-node traffic fails. -
Default value:
16384(can accommodate approximately 16,000 unique node IP addresses). -
Limitation: The source code hard-codes that a value lower than the default of 16384 is not allowed. At startup, if
bpf-node-map-max < 16384, the agent immediately exits with an error. The upper limit is constrained byuint32(theoretical maximum4,294,967,295), but the actual limit depends on the node memory. -
When to adjust:
-
The cluster size is approaching 16,000 nodes, or features that inflate the number of entries per node, such as IPsec/WireGuard, are enabled.
-
The metric
cilium_bpf_map_pressure{map_name="cilium_node_map_v2"}shows that the pressure is >0.8.
-
-
How to adjust: In
cilium_args, append--bpf-node-map-max=32768, and restart the terway-eniip pods for the change to take effect.
If a node has dual stack enabled or obtains other addresses, a single node can occupy two or three entries.
bpf-lb-map-max
-
Function: Determines the data plane programming capacity for Kubernetes Services across the entire cluster. It covers all three Service types (ClusterIP, NodePort, and LoadBalancer) and their LB sub-maps, such as backends, reverse NAT, session affinity, and Maglev hash tables. When the map is full, newly created Services or newly added backend pods cannot be programmed into BPF, which appears as unreachable Service IP addresses or delayed endpoint updates.
-
Scope: Cilium uses the LB Service Maps named
cilium_lb{4,6}_services_v2to store the load balancing entries for ClusterIP and NodePort Services. This parameter also serves as the default upper limit for the following LB sub-maps (if not separately specified):-
cilium_lb4_services_v2/cilium_lb6_services_v2 -
cilium_lb4_backends_v3/cilium_lb6_backends_v3 -
cilium_lb4_reverse_nat/cilium_lb6_reverse_nat -
cilium_lb4_affinity/cilium_lb6_affinity -
cilium_lb4_source_range/cilium_lb6_source_range
-
-
Default value:
65536(64Ki). If this map is full, Cilium may fail to update Service endpoints, which affects the connectivity of Service IP addresses or the ability to create new Services.
The size required for the Service LB map depends on the number of Services, the backend pods behind each Service, and the port/protocol combinations each Service defines.
Each ClusterIP/NodePort Service creates entries equal to the number of its backend pods multiplied by the number of port and protocol entries in the corresponding Service spec.
Based on this, estimate the required map size as follows:
Note that this estimation assumes that the number of pods selected by each Service and the number of port and protocol entries are roughly normally distributed. If your use case has significant outliers (for example, a Service that selects a very large number of backend pods), you may need a more detailed estimation.
Adjusting the bpf-lb-map-max parameter and restarting Cilium interrupts connections because the new map must repopulate the existing Service entries.