Configure exclusive ENI mode for a node pool

1. Plan the exclusive ENI node pool

• In exclusive ENI mode, each node supports fewer pods and requires more than six ENIs. See instance family for ENI counts per instance type.

• Plan the vSwitches and security groups for your pods.

Terway supports the following configuration methods in descending priority:

• Configure a fixed IP address, an independent virtual switch, and a security group for a pod

• Node-level network configuration

• Cluster default configuration: Customize Terway configuration parameters

  Important

  Include vSwitches for each node's zone. Otherwise, pod creation fails.

  Lingjun node pools do not support configuring a fixed IP address, an independent vSwitch, and a security group for a pod.

2. Create an exclusive ENI node pool and verify the configuration

1. Follow Create and manage node pools to create a node pool, and add the label k8s.aliyun.com/exclusive-mode-eni-type: eniOnly to nodes.

   Also add a taint to prevent pods that do not require an exclusive ENI from being scheduled to this pool.

   Important

   Add the label during node pool creation. Existing nodes cannot switch to exclusive ENI mode. If the label is incorrect, delete and recreate the node pool.

2. Query the allocatable resources on a node to verify that exclusive ENI mode is enabled:

   kubectl describe node <node-name> 

   Expected output:

   Capacity:
     aliyun/eni:         7
     cpu:                16
     ephemeral-storage:  123460788Ki
     hugepages-1Gi:      0
     hugepages-2Mi:      0
     memory:             31555380Ki
     pods:               213
   Allocatable:
     aliyun/eni:         7
     cpu:                15890m
     ephemeral-storage:  113781462033
     hugepages-1Gi:      0
     hugepages-2Mi:      0
     memory:             28587828Ki
     pods:               213

   The aliyun/eni field confirms that exclusive ENI mode is enabled.

3. Schedule pods to the exclusive ENI node pool

Use NodeAffinity or a PodNetworking custom resource to schedule pods to the exclusive ENI node pool.

• NodeAffinity: Does not support pod-level configurations such as fixed IP, independent vSwitch, or security group.

• PodNetworking: Supports pod-level vSwitch, security group, and fixed IP configurations. See Configure a fixed IP address, an independent virtual switch, and a security group for a pod.

  Set eniType to ENI in the eniOptions field to schedule pods to the exclusive ENI node pool. Example:

  apiVersion: network.alibabacloud.com/v1beta1
  kind: PodNetworking
  metadata:
    name: enionly
  spec:
    eniOptions:
      eniType: ENI
    allocationType:
      type: Elastic
    selector:
      podSelector:
        matchLabels:
          network: enionly

Verifying a pod's exclusive ENI

Terway automatically creates a PodENI resource with the same name and namespace as the pod to record its network configuration.

Query the resource:

kubectl get podeni nginx-9d557694f-rcdzs -oyaml

Expected output:

apiVersion: network.alibabacloud.com/v1beta1
kind: PodENI
metadata:
  annotations:
    k8s.aliyun.com/pod-uid: 05590939-fc51-47ab-a204-3dd187233bca
  creationTimestamp: "2024-09-13T08:09:27Z"
  finalizers:
  - pod-eni
  generation: 1
  labels:
    k8s.aliyun.com/node: cn-hangzhou.172.XX.XX.25
  name: example-9d557694f-rcdzs
  namespace: default
  resourceVersion: "1131123"
spec:
  allocations:
  - allocationType:
      type: Elastic
    eni:
      attachmentOptions: {}
      id: eni-xxxx
      mac: 00:16:3e:37:xx:xx
      securityGroupIDs:
      - sg-xxxx
      vSwitchID: vsw-xxxx
      zone: cn-hangzhou-j
    ipv4: 172.16.0.30
    ipv4CIDR: 172.16.0.0/24
    ipv6: 2408:4005:xxxx:xxxx:xxxx:xxxx:xxxx:9ad4
    ipv6CIDR: 2408:4005:39c:xxxx::/64
  zone: cn-hangzhou-j
status:
  eniInfos:
    eni-xxxx:
      id: eni-xxxx
      status: Bind
      type: Secondary
  instanceID: i-xxxx
  phase: Bind