By default, Pods in a Kubernetes cluster use private IP addresses. However, some scenarios require a Pod to have an independent public IP address to communicate with external networks, unaffected by traffic from other Pods. Alibaba Cloud's Terway and ack-extend-network-controller components allow you to associate an elastic IP address (EIP) directly with a Pod, giving it an independent public egress.
Prerequisites
You must have an ACK Managed cluster or an ACK Dedicated cluster that uses the Terway network plugin. For instructions, see Create an ACK Managed cluster and Create an ACK Dedicated cluster (Creation of new clusters is no longer supported). For more information about the Terway network mode, see Use the Terway network plugin.
As of November 2023, the EIP feature in Terway is no longer maintained and was removed in v1.7.0. Use the ack-extend-network-controller component to attach independent EIPs to Pods. For more information, see Migrate the EIP feature from Terway to ack-extend-network-controller.
Limits
-
Before using an EIP, review its limits. For more information, see Limits.
-
This feature is supported only on ECS nodes.
-
The component manages the lifecycle of an EIP created by the controller. If you uninstall the component after an EIP is assigned to a Pod, the EIP remains allocated and is not automatically released.
To stop using the EIP feature, you must first delete the Pods that are using EIPs before you stop the component. You can check the allocation and release status of an EIP by inspecting the
podeips.alibabacloud.comresource with the same name as the Pod. Ensure the EIP is no longer in use before you stop the component. Otherwise, billing for the EIP will continue.
Billing
While the Terway and ack-extend-network-controller plugins are free to enable, charges apply for the cloud resources they consume. For more information about ACK billing, see Billing for cloud resources.
Background
In most cases, Pods in Terway network mode access the public network using host SNAT (Source Network Address Translation) or external SNAT through an EIP. Typically, a Service of type LoadBalancer routes public ingress traffic to Pods. However, some scenarios require a Pod to have an independent public IP address. For example:
-
A Pod exposes services on random ports. This is common for UDP (User Datagram Protocol)-based applications like game servers or teleconferencing. For example, the RTSP protocol uses a different port for each client.
-
A Pod requires a dedicated public egress to avoid contention with other Pods.
Step 1: Configure RAM permissions for EIPs
ack-extend-network-controller (Recommended)
-
Grant RAM permissions to an ACK managed cluster or an ACK dedicated cluster
-
Create a custom permission policy with the following content. For instructions, see Step 1: Create a custom permission policy.
{ "Effect": "Allow", "Action": [ "vpc:DescribeVSwitches", "vpc:AllocateEipAddress", "vpc:AllocateEipAddressPro", "vpc:DescribeEipAddresses", "vpc:AssociateEipAddress", "vpc:UnassociateEipAddress", "vpc:ReleaseEipAddress", "vpc:AddCommonBandwidthPackageIp", "vpc:RemoveCommonBandwidthPackageIp", "vpc:TagResources", "ecs:DescribeNetworkInterfaces" ], "Resource": [ "*" ], "Condition": {} } -
Grant permissions to the worker RAM role of the cluster. For instructions, see Step 2: Grant permissions to the worker RAM role of the cluster.
-
Terway
Terway requires EIP-related permissions to configure EIPs for pods. To do so, you must deploy additional configurations and grant permissions.
-
Grant RAM permissions to an ACK managed cluster or an ACK basic managed cluster created in or after June 2020
-
Create a custom permission policy with the following content. For instructions, see Step 1: Create a custom permission policy.
{ "Effect": "Allow", "Action": [ "vpc:DescribeVSwitches", "vpc:AllocateEipAddress", "vpc:DescribeEipAddresses", "vpc:AssociateEipAddress", "vpc:UnassociateEipAddress", "vpc:ReleaseEipAddress", "vpc:AddCommonBandwidthPackageIp", "vpc:RemoveCommonBandwidthPackageIp" ], "Resource": [ "*" ], "Condition": {} } -
Grant permissions to the worker RAM role of the cluster. For instructions, see Step 2: Grant permissions to the worker RAM role of the cluster.
-
-
Grant RAM permissions to an ACK dedicated cluster or an ACK managed cluster created before June 2020
-
Create a custom permission policy with the following content. For instructions, see Step 1: Create a custom permission policy.
{ "Effect": "Allow", "Action": [ "vpc:DescribeVSwitches", "vpc:AllocateEipAddress", "vpc:DescribeEipAddresses", "vpc:AssociateEipAddress", "vpc:UnassociateEipAddress", "vpc:ReleaseEipAddress", "vpc:AddCommonBandwidthPackageIp", "vpc:RemoveCommonBandwidthPackageIp" ], "Resource": [ "*" ], "Condition": {} } -
On the Policies tab of the AliyunCSManagedNetworkRole RAM role page, click Grant Permission. Then, click Custom Policy, select the custom permission policy you created, and click OK.
-
Step 2: Install or upgrade a cluster plugin
Do not enable the EIP feature in both ack-extend-network-controller and Terway simultaneously. This may cause duplicate Elastic IP Address (EIP) allocations and result in binding failures. Use the ack-extend-network-controller plugin.
Plugin comparison
|
Item |
ack-extend-network-controller |
Terway |
|
Supported cluster types |
|
|
|
Static EIP |
Supported |
Not supported |
|
Configuration phase |
After a pod IP is assigned, the controller assigns and binds an EIP to the pod. |
Assigns and binds an EIP during the CNI execution process. |
|
Supported versions |
Plugin version v0.2.0 or later. |
Terway plugin version v1.0.10.280-gdc2cb6c-aliyun or later, but earlier than v1.7.0. |
Install or upgrade a plugin
If you use the ack-extend-network-controller plugin, install it from the Marketplace and enable the EIP controller. If you use the Terway network plugin, upgrade it to version v1.0.10.280-gdc2cb6c-aliyun or later.
ack-extend-network-controller (recommended)
Log on to the ACK console. In the left navigation pane, click .
-
On the Marketplace page, enter
ack-extend-network-controllerin the search bar, and then click the target application. -
On the application details page, click Deploy in the upper-right corner.
-
In the Create panel, select a cluster and namespace, and then click Next.
-
On the OK page, select a version, configure the parameters, and then click OK.
The following table describes the parameters.
Parameter
Type
Required
Description
clusterID
string
Yes
The cluster ID.
regionID
string
Yes
The cluster region.
enableControllers
[]string
Yes
Configure
eipto enable the EIP feature.networkController.vpcID
string
Yes
The ID of the cluster's Virtual Private Cloud (VPC).
networkController.eipController.maxConcurrentReconciles
int
No
The number of concurrent reconciles for the EIP controller.
networkController.eipController.garbageCollectionPeriodInMinutes
int
No
The interval, in minutes, for static EIP garbage collection.
networkController.customStatefulWorkloadKinds
[]string
No
The kinds of custom stateful controllers.
enableVirtualNode
bool
No
Enables support for Elastic Container Instance (ECI) virtual nodes.
ImportantIf you enable this feature, the controller assigns EIPs to pods on ECI instances. The configured pod annotation cannot be the same as the one used for the EIP feature supported by ECI. Use this feature with caution.
enableRRSA
bool
No
Enables RRSA for authentication.
NoteSupported by ack-extend-network-controller plugin v0.10.0 and later.
RRSA configuration supports only the ack-pod-identity-webhook injection method. For configuration details, see Use RRSA to configure RAM permissions for a ServiceAccount to isolate pod permissions.
rrsaRoleName
string
No
The role name.
NoteSupported by ack-extend-network-controller plugin v0.10.0 and later.
credential.accessKey
string
No
The AccessKey ID of the account with permissions to attach EIPs.
This parameter is not required if you completed the setup in Step 1: Configure the RAM permissions required to attach an EIP.
ImportantNot recommended, as this method stores sensitive information in a Kubernetes Secret.
credential.accessSecret
string
No
The AccessKey Secret of the account with permissions to attach EIPs.
This parameter is not required if you completed the setup in Step 1: Configure the RAM permissions required to attach an EIP.
ImportantNot recommended, as this method stores sensitive information in a Kubernetes Secret.
kubeClientQPS
float32
No
The queries per second (QPS) limit for the component when accessing the cluster API server.
NoteSupported by ack-extend-network-controller plugin v0.12.2 and later.
kubeClientBurst
int
No
The maximum burst requests from the component to the cluster API server.
NoteSupported by ack-extend-network-controller plugin v0.12.2 and later.
The following code provides a sample configuration:
clusterID: "c11ba338192xxxxxxx" regionID: "cn-hangzhou" vpcID: "vpc-bp1rkq0zxxxxxx" enableControllers: - eip networkController: eipController: maxConcurrentReconciles: 1 garbageCollectionPeriodInMinutes: 1 customStatefulWorkloadKinds: - foo credential: accessKey: "" accessSecret: ""
-
To update the version and parameters of the ack-extend-network-controller plugin, see Modify a Helm chart.
Terway
If your cluster uses the Terway network plugin, upgrade it (terway-eni or terway-eniip) to a version that supports the EIP feature. For detailed instructions, see Manage components.
The Terway plugin version must be v1.0.10.280-gdc2cb6c-aliyun or later, but earlier than v1.7.0. For details about Terway versions, see Terway.
Step 3: Enable the EIP feature
EIP annotations
ACK (Container Service for Kubernetes) lets you enable the EIP (Elastic IP Address) feature by using annotations. You can automatically assign an EIP to a pod or associate an existing EIP with one.
Automatically assign an EIP
When you automatically assign EIPs, the system may repeatedly create and release them during pod recreation or after CNI execution failures. To avoid this, specify an existing EIP instance instead.
|
Pod annotations |
Value |
|
network.alibabacloud.com/pod-with-eip k8s.aliyun.com/pod-with-eip |
Specifies whether to automatically create and associate an EIP. Valid values:
|
|
k8s.aliyun.com/eci-with-eip (Compatible) |
|
|
network.alibabacloud.com/eip-bandwidth k8s.aliyun.com/eip-bandwidth |
This annotation applies only when creating an EIP. Specifies the peak bandwidth of the EIP in Mbps. For more information, see Request an EIP. |
|
network.alibabacloud.com/eip-internet-charge-type k8s.aliyun.com/eip-internet-charge-type |
This annotation applies only when creating an EIP. Specifies the metering method of the EIP. Valid values:
For more information, see Request an EIP. |
|
k8s.aliyun.com/eip-charge-type (Compatible) |
|
|
network.alibabacloud.com/eip-instance-charge-type k8s.aliyun.com/eip-instance-charge-type |
This annotation applies only when creating an EIP. Specifies the billing method of the EIP. Valid values:
For more information, see Request an EIP. Note
This annotation is supported only by ack-extend-network-controller. The PrePaid (subscription) option is not supported. |
|
network.alibabacloud.com/eip-common-bandwidth-package-id k8s.aliyun.com/eip-common-bandwidth-package-id |
Specifies the ID of an EIP Bandwidth Plan to associate with the EIP. Note
This feature requires Terway v1.2.3 or later. No version restrictions apply to the ack-extend-network-controller component. |
|
network.alibabacloud.com/eip-isp k8s.aliyun.com/eip-isp |
This annotation applies only when creating an EIP. Specifies the line type of the EIP. Valid values:
For more information, see Request an EIP. Note
This feature requires Terway v1.2.3 or later. No version restrictions apply to the ack-extend-network-controller component. |
|
network.alibabacloud.com/eip-public-ip-address-pool-id k8s.aliyun.com/eip-public-ip-address-pool-id |
This annotation applies only when creating an EIP. Specifies the ID of the IP address pool. For information about the limits and procedures for using IP address pools, see IP address pools. Note
This is supported only by ack-extend-network-controller. |
|
network.alibabacloud.com/eip-resource-group-id k8s.aliyun.com/eip-resource-group-id |
This annotation applies only when creating an EIP. Specifies the ID of the resource group for the EIP. For more information, see Request an EIP. Note
This is supported by ack-extend-network-controller v0.4.0 and later. |
|
network.alibabacloud.com/eip-name k8s.aliyun.com/eip-name |
This annotation applies only when creating an EIP. Specifies the name of the EIP. For more information, see Request an EIP. Note
This is supported by ack-extend-network-controller v0.6.0 and later. |
|
network.alibabacloud.com/eip-description k8s.aliyun.com/eip-description |
This annotation applies only when creating an EIP. Specifies the description of the EIP. For more information, see Request an EIP. Note
This is supported by ack-extend-network-controller v0.6.0 and later. |
|
network.alibabacloud.com/eip-security-protection-types k8s.aliyun.com/eip-security-protection-types |
This annotation applies only when creating an EIP. Specifies the security protection level for the EIP. Anti-DDoS enabled EIPs are supported.
To specify multiple protection levels, separate them with a comma ( Note
This is supported by ack-extend-network-controller v0.9.0 and later. |
|
network.alibabacloud.com/eip-tags k8s.aliyun.com/eip-tags |
This annotation applies only when creating an EIP. Specifies the tags to add to the EIP. The value must be a valid JSON string. For more information, see Manage tags. Example: Note
This is supported by ack-extend-network-controller v0.11.0 and later. |
Specify an EIP instance
You can associate an existing EIP with a pod by specifying the EIP instance ID. The pod annotation does not modify the EIP instance configuration. It only associates the EIP with the specified pod.
This feature is not suitable for controllers with multiple replicas, such as Deployments. Ensure that only one pod references the EIP instance. We recommend that you use this feature only with stateful applications, such as StatefulSets.
|
Pod annotations |
Value |
|
network.alibabacloud.com/pod-eip-instanceid k8s.aliyun.com/pod-eip-instanceid |
The ID of the EIP instance to associate, for example, eip-bp14qxxxxxxx. Important
Do not assign the same EIP instance to pods with different names. After a pod is deleted, the EIP controller disassociates the EIP. During this period, you cannot use the same EIP instance for a new pod. To check whether the disassociation is complete, verify that the PodEIP resource that has the same name as the pod no longer exists. |
|
k8s.aliyun.com/eci-eip-instanceid (Compatible) |
EIP release strategy annotations
This release strategy allows a pod to retain its EIP address even after being recreated. You can use this strategy with the automatic assignment feature to provide a persistent EIP for stateful applications.
-
This feature is supported only by ack-extend-network-controller and applies only to stateful replica controllers. It cannot be used for stateless controllers.
-
If you specify an EIP instance ID, the EIP instance is not released when the pod is deleted.
|
Pod annotation |
Value |
|
network.alibabacloud.com/pod-eip-release-strategy k8s.aliyun.com/pod-eip-release-strategy |
The release strategy for the pod's EIP. Valid values:
You can also specify a retention period. Go-style duration expressions are supported. For example, |
Enable EIP in ack-extend-network-controller or Terway
Enable in ack-extend-network-controller
The ack-extend-network-controller component calls Alibaba Cloud APIs to create resources. You must configure the required permissions in RAM (Resource Access Management), install ack-extend-network-controller from the Marketplace, and then use annotations to create and associate EIPs with specific pods.
Create and associate EIPs with annotations
You can use annotations in a pod's manifest to create an EIP and associate it with that pod. For more information about annotations, see FAQ.
-
Create an application, and then use annotations to create and associate an EIP with a specific pod.
Deployment
This manifest creates a Deployment where ACK automatically assigns an EIP instance with a peak bandwidth of 5 Mbps to each pod.
apiVersion: apps/v1 kind: Deployment metadata: name: example labels: app: example spec: replicas: 1 selector: matchLabels: app: example template: metadata: labels: app: example annotations: k8s.aliyun.com/pod-with-eip: "true" k8s.aliyun.com/eip-bandwidth: "5" spec: readinessGates: - conditionType: "k8s.aliyun.com/eip" containers: - name: example image: nginxAfter the pod is created, query the
podeips.alibabacloud.comresource that has the same name as the pod to view details of the assigned EIP.kubectl get podeip -n <namespace> <podname> -o yamlExpected output:
apiVersion: alibabacloud.com/v1beta1 kind: PodEIP metadata: creationTimestamp: "2023-12-15T04:25:37Z" finalizers: - podeip-controller.alibabacloud.com/finalizer generation: 1 name: example-xxx namespace: default resourceVersion: "222800" uid: 43xxx-f1xx-4xxx-b3xx-969faxxx spec: allocationID: eip-2ze2qe8zsxxx allocationType: releaseStrategy: Follow type: Auto status: eipAddress: 39.102.XX.XX internetChargeType: PayByTraffic isp: BGP networkInterfaceID: eni-2zeagv8f3xxxx podLastSeen: "2023-12-15T05:18:47Z" privateIPAddress: 192.168.XX.XX resourceGroupID: rg-acfmwxxxxxsq status: InUseStatefulSet
Use the following manifest to create a StatefulSet with two pods. An EIP is automatically assigned to each pod, and the release strategy is set to delete the PodEIP resource 10 minutes after the corresponding pod is deleted.
apiVersion: apps/v1 kind: StatefulSet metadata: name: example labels: app: example spec: serviceName: "example" replicas: 2 selector: matchLabels: app: example template: metadata: labels: app: example annotations: k8s.aliyun.com/pod-with-eip: "true" k8s.aliyun.com/pod-eip-release-strategy: "10m" spec: containers: - name: example image: nginxAfter the pods are created, query the
podeips.alibabacloud.comresources to track information about the assigned EIPs.kubectl get podeip -n <namespace> -o yamlExpected output:
apiVersion: v1 items: - apiVersion: alibabacloud.com/v1beta1 kind: PodEIP metadata: creationTimestamp: "2023-12-15T03:28:01Z" finalizers: - podeip-controller.alibabacloud.com/finalizer generation: 1 name: example-0 namespace: default resourceVersion: "227221" uid: 79954xx-17xx-4dxx-b7xx-15b84xxx spec: allocationID: eip-2ze08metxxx allocationType: releaseAfter: 10m releaseStrategy: TTL type: Auto status: eipAddress: 39.105.XX.XX internetChargeType: PayByTraffic isp: BGP networkInterfaceID: eni-2ze4tkg4xxx podLastSeen: "2023-12-15T05:31:34Z" privateIPAddress: 192.168.XX.XX resourceGroupID: rg-acfmwxxx status: InUse - apiVersion: alibabacloud.com/v1beta1 kind: PodEIP metadata: creationTimestamp: "2023-12-15T03:28:03Z" finalizers: - podeip-controller.alibabacloud.com/finalizer generation: 1 name: example-1 namespace: default resourceVersion: "227222" uid: 1339xxxe7-97xx-46xx-9bxx-537690xxx spec: allocationID: eip-2zetwhffqxxx allocationType: releaseAfter: 10m releaseStrategy: TTL type: Auto status: eipAddress: 39.105.XX.XX internetChargeType: PayByTraffic isp: BGP networkInterfaceID: eni-2zeagv8f3wxxx podLastSeen: "2023-12-15T05:31:34Z" privateIPAddress: 192.168.XX.XX resourceGroupID: rg-acfmwqnwxxx status: InUse - apiVersion: alibabacloud.com/v1beta1 kind: PodEIP metadata: creationTimestamp: "2023-12-15T04:25:37Z" finalizers: - podeip-controller.alibabacloud.com/finalizer generation: 1 name: example-5bxxx-9xx namespace: default resourceVersion: "227220" uid: 43cdfxxx-f1xx-42xx-b3xx-969fxxx spec: allocationID: eip-2ze2qe8zsmnxxx allocationType: releaseStrategy: Follow type: Auto status: eipAddress: 39.102.XX.XX internetChargeType: PayByTraffic isp: BGP networkInterfaceID: eni-2zeagv8f3wxxx podLastSeen: "2023-12-15T05:31:34Z" privateIPAddress: 192.168.XX.XX publicIpAddressPoolID: pippool-2ze498cxxx resourceGroupID: rg-acfmwqnxxx status: InUse kind: List metadata: resourceVersion: ""After a pod in this stateful application is deleted, its PodEIP custom resource (CR) is retained for 10 minutes before being deleted. If a pod with the same name is created during this period, it reuses the corresponding EIP.
-
Verify the configuration.
After the pod enters the Running state, check the value of the k8s.aliyun.com/allocated-eipAddress annotation on the pod to view its assigned EIP address. You can access the pod by using this EIP.
Enable in Terway
-
Deploy the Terway configuration to support the EIP feature.
-
Edit the Terway ConfigMap.
kubectl edit cm eni-config -n kube-system -
In eni_conf, add the following line.
"enable_eip_pool": "true"NoteTo forcibly disassociate an EIP from its current instance when you associate it with a new one, add the "allow_eip_rob": "true" parameter to eni_conf.
-
After you finish editing, press the Esc key, enter :wq!, and press the Enter key to save your changes and exit edit mode.
-
Recreate the Terway pods to apply the changes.
kubectl delete pod -n kube-system -l app=terway-eniip # If you installed the terway-eni component, replace terway-eniip with terway-eni.
-
-
Create and associate an EIP with a pod by using annotations.
You can use annotations in a pod's manifest to create an EIP and associate it with that pod. For more information, see EIP-related limits.
Automatically assign an EIP
Add the following annotations to automatically assign an EIP to a pod and specify the EIP bandwidth.
-
k8s.aliyun.com/pod-with-eip: "true": Assign a dedicated public EIP to the target pod. -
k8s.aliyun.com/eip-bandwidth: "5": Specify the bandwidth of the EIP. The default is 5 Mbps, which is consistent with the EIP default value.
The following is a sample manifest.
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment-basic labels: app: nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: annotations: network.alibabacloud.com/pod-with-eip: "true" # Automatically assign a public EIP to the Nginx container. network.alibabacloud.com/eip-bandwidth: "5" # Specify the bandwidth of the EIP. The default is 5 Mbps, which is consistent with the EIP default value. labels: app: nginx spec: containers: - name: nginx image: nginxSpecify an EIP
Important-
Because a single EIP cannot be associated with multiple pods, this method is not suitable for workloads such as Deployments and StatefulSets that have more than one replica.
-
By default, if the specified EIP is already associated with an instance, the association fails. To force the disassociation of the EIP from the previous instance and associate it with the new one, you must set "allow_eip_rob": "true" in the ConfigMap as described earlier.
-
You can specify an EIP instance ID only for single-replica workloads. For example, a Deployment cannot have more than one replica.
Add the following annotation to specify an EIP instance ID for a pod:
k8s.aliyun.com/pod-eip-instanceid: "<your-eip-instance-id>" -
-
Verify the configuration.
After the pod enters the Running state, check the value of the network.alibabacloud.com/allocated-eipAddress annotation on the pod to view its assigned EIP address. You can access the pod by using this EIP.
FAQ
Pod ready before EIP is bound
Because the controller assigns an EIP to a Pod only after its IP address is allocated, the Pod can become Ready before the EIP is bound. Use the following methods to control this timing.
Readiness gates
Configuring readiness gates requires the ack-extend-network-controller.
When you configure readinessGates for a Pod, the controller updates the Pod's conditions only after the EIP is bound. This prevents the Pod from becoming Ready until the EIP is bound.
kind: Pod
...
spec:
readinessGates:
- conditionType: "k8s.aliyun.com/eip"
status:
conditions:
- lastProbeTime: "2022-12-12T03:45:48Z"
lastTransitionTime: "2022-12-12T03:45:48Z"
reason: Associate eip succeed
status: "True"
type: k8s.aliyun.com/eip
...
Init container
You can configure an init container to check if an EIP is assigned to the Pod.
apiVersion: v1
kind: Pod
metadata:
name: example
annotations:
network.alibabacloud.com/pod-with-eip: "true"
spec:
containers:
- name: example
image: busybox:1.28
command: ['sh', '-c', 'echo The app is running! && sleep 3600']
initContainers:
- name: init
image: busybox:1.28
command: ['timeout', '-t' ,'60', 'sh','-c', "until grep -E '^k8s.aliyun.com\\/allocated-eipAddress=\\S?[0-9]+\\S?' /etc/podinfo/annotations; do echo waiting for annotations; sleep 2; done"]
volumeMounts:
- name: podinfo
mountPath: /etc/podinfo
volumes:
- name: podinfo
downwardAPI:
items:
- path: "labels"
fieldRef:
fieldPath: metadata.labels
- path: "annotations"
fieldRef:
fieldPath: metadata.annotations
Pod inaccessible via EIP
Restrictive ACL policies often cause this issue. Check the following access control settings:
-
Cloud Firewall: Traffic to an EIP is subject to the access control rules of Cloud Firewall. Ensure you have rules that allow inbound and outbound traffic for the Pod's EIP and its service ports. For more information, see Configure access control policies for an internet firewall.
-
Node security group: After an EIP is attached to a Pod, the security group of the Pod's node continues to govern traffic. Ensure the
node security grouprules allow inbound traffic from the client IP address to the Pod's service ports, as well as any necessary outbound traffic. For more information, see Configure cluster security groups.