Best practices for Terway Datapath V2

Enable Prometheus metrics

Datapath V2 reuses the Cilium Agent metric endpoint. Add the --prometheus-serve-addr argument to cilium_args in the Terway ConfigMap:

"cilium_args": "--prometheus-serve-addr=:9962"

• :9962 is the Cilium convention for the metrics port (set to empty string to disable). Listens on 0.0.0.0, allowing direct scraping from the pod's hostNetwork port.

• Restart the terway-eniip pod for the changes to take effect.

• Add scrape annotations to your pod or service, or create a ServiceMonitor for automatic Prometheus discovery:

  metadata:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9962"
      prometheus.io/path: "/metrics"

Key BPF map metrics

Some metrics require Terway v1.14.0 or later. Upgrade to the latest version.

Alerting and query examples

# Alert if any BPF map usage exceeds 80% for 5 minutes
max by (map_name) (cilium_bpf_map_pressure) > 0.8

# Each node's LB Service Map usage
cilium_bpf_map_pressure{map_name=~"cilium_lb[46]_services_v2"}

# Each endpoint's Policy Map usage (Top 10)
topk(10, cilium_bpf_map_pressure{map_name=~"cilium_policy_.*"})

# Node Map capacity alert (increase bpf-node-map-max if >80%)
cilium_bpf_map_pressure{map_name="cilium_node_map_v2"} > 0.8

# Global Conntrack Map pressure (related to bpf-map-dynamic-size-ratio tuning)
cilium_bpf_map_pressure{map_name=~"cilium_ct.*_global"} > 0.7

# Map write failure rate
sum by (map_name) (rate(cilium_bpf_map_ops_total{operation="update",outcome="fail"}[5m])) > 0

If cilium_bpf_map_pressure consistently exceeds 0.8, adjust the corresponding parameters as described in the bpf map capacity tuning section. A value exceeding 0.95 is an emergency requiring immediate capacity expansion to prevent entry programming failures.

bpf-map-dynamic-size-ratio

• Function: Controls the number of concurrent connections to track and NodePort/SNAT translation entries per node. Capacity is dynamically calculated from node physical memory, this ratio, and entry size. When full, new connections fail and existing ones may be evicted, causing packet loss.

• Scope: The capacity of several global maps — CT (cilium_ct4_global / cilium_ct_any4_global), NAT (cilium_snat_v4_external), Neighbor (cilium_nodeport_neigh4), and SockRevNAT — is dynamically calculated from this parameter and node physical memory.

• Value range: (0.0, 1.0], with a default of 0.0025 (approximately 0.25% of physical memory).

• Typical calculation (based on Cilium default element sizes):

  Node physical memory	CT TCP entries	CT any entries	NAT entries
  512 MiB	33,140	16,570	33,140
  1 GiB	66,280	33,140	66,280
  4 GiB	265,121	132,560	265,121
  16 GiB	1,060,485	530,242	1,060,485

• Safeguards: Results are clamped to [1Ki, 16Mi] (approximately 1 GiB of memory per map). Maximum entries per map: 2^24 = 16,777,216.

• Tuning recommendations:

  • For high-concurrency nodes (over 10,000 long-lived connections per node), increase to 0.005–0.01 to allow more CT/NAT entries on large-memory nodes.

  • Maps with explicitly set bpf-ct-global-tcp-max, bpf-ct-global-any-max, or bpf-nat-global-max are not affected.

  • If monitoring bpf map usage shows that cilium_bpf_map_pressure for maps such as cilium_ct4_global consistently exceeds 0.8, adjust this ratio or explicitly set the capacity for the affected map.

bpf-policy-map-max

• Scope: Each pod has its own cilium_policy_* map storing allowed combinations of peer identity, port, and protocol for data plane NetworkPolicy enforcement.

• Default value: 16384. minimum value: 256 (2^8), maximum value: 65536 (2^16). Out-of-range values are clamped with a warning.

• When to adjust:

  • NetworkPolicy is enabled in the cluster, and the number of identities is large (see Limit the number of identities).

  • The product of peer identities and exposed port/protocol combinations for a single pod approaches 16384.

  • The metric cilium_bpf_map_pressure{map_name=~"cilium_policy_.*"} consistently exceeds 0.8.

• How to adjust: Add --bpf-policy-map-max=32768 to cilium_args in the Terway ConfigMap and restart the terway-eniip pod.

• Note: Memory per map is proportional to max_entries, and each node hosts one map per pod. Setting 65536 without assessment significantly increases node memory overhead.

bpf-node-map-max

• Function: Controls the maximum number of nodes the cluster can recognize. Stores Node ID-to-IP mappings in a single global map for cross-node communication, IPsec/WireGuard encryption, and Egress Gateway. When full, new nodes cannot be recognized, causing cross-node traffic failures.

• Scope: A single cilium_node_map_v2 map shared across the cluster for cross-node communication, encryption, and Egress Gateway.

• Default value: 16384 (can hold approximately 16,000 unique node IPs).

• Limitation: The source code prohibits setting a value lower than the default of 16384. If bpf-node-map-max < 16384 at startup, the agent exits with an error. Upper limit: uint32 (theoretically 4,294,967,295), constrained by node memory.

• When to adjust:

  • The cluster size is approaching 16,000 nodes, or features that increase per-node entries, such as IPsec/WireGuard, are enabled.

  • The metric cilium_bpf_map_pressure{map_name="cilium_node_map_v2"} consistently exceeds 0.8.

• How to adjust: Add --bpf-node-map-max=32768 to cilium_args and restart the terway-eniip pod.

bpf-lb-map-max

• Function: Controls the data plane capacity for all Kubernetes services in the cluster — ClusterIP, NodePort, and LoadBalancer — including backends, reverse NAT, session affinity, and Maglev hash tables. When full, new services or backends cannot be programmed, causing unreachable service IPs or delayed endpoint updates.

• Scope: Sets the capacity for cilium_lb{4,6}_services_v2 (ClusterIP and NodePort load balancing entries) and the following LB sub-maps (unless individually overridden):

  • cilium_lb4_services_v2 / cilium_lb6_services_v2

  • cilium_lb4_backends_v3 / cilium_lb6_backends_v3

  • cilium_lb4_reverse_nat / cilium_lb6_reverse_nat

  • cilium_lb4_affinity / cilium_lb6_affinity

  • cilium_lb4_source_range / cilium_lb6_source_range

• Default value: 65536 (64K). When full, Cilium cannot update service endpoints, affecting connectivity or new service creation.

• Capacity estimation: Entries per service = backend pods × port/protocol combinations in the service spec.

Estimate the required LB map size:

Entries per ClusterIP/NodePort service = backend pods selected by the service × port/protocol entries in the Service spec.

$$\text{Entries/Service}=(\text{BackendPods})\times (\text{Port/Protocol Combos})$$

Estimated total map size:

$$\text{Total LB Entries}\approx (\text{services})\times (\text{Avg. Endpoints/Service})\times (\text{Avg. Port/Protocol Combos/Service})$$

This assumes a roughly normal distribution of pods and port/protocol entries per service. For significant outliers (for example, one service with many backends), perform a more detailed estimation.