Terway Hybrid network plugin-Container Service for Kubernetes(ACK)-阿里云帮助中心

Connecting a hybrid cloud node pool to an on-premises data center creates complex network topologies and cross-domain routing requirements that can exceed the capabilities of standard container network plugins. The Terway Hybrid network plugin is specifically designed for hybrid cloud node pools and ensures network connectivity between all pods in a cluster, regardless of whether they are running in the cloud or in an on-premises data center.

How it works

The following figure shows the overall network architecture of an ACK hybrid cloud node pool. The architecture consists of two main network domains: an Alibaba Cloud VPC and an on-premises data center. If you have multiple on-premises data centers or connect to third-party clouds, you may have more network domains. These network domains use a dedicated connection to establish Layer 3 private network connectivity. The ACK cluster in the VPC can use a standard network plugin, such as Flannel or Terway. The nodes in the on-premises data center must use the Terway Hybrid network plugin.

Terway Hybrid modes

Terway Hybrid provides two modes at the node pool level. You can configure the mode when you create a hybrid cloud node pool.

Comparison	Underlay mode	Overlay mode
Advantages	High performance with low network latency due to no VXLAN encapsulation overhead. Performance is about 20% higher than in overlay mode.	Simple configuration with no special requirements for the underlying network, offering greater deployment flexibility.
Network requirements	Requires Layer 2 network connectivity between nodes. If nodes are in different Layer 3 network domains, you must configure BGP to advertise pod routes.	Requires only Layer 3 private network connectivity between nodes.
pod network path	Pod traffic is routed at Layer 3 directly through the host network interface card (NIC).	Pod traffic is encapsulated using VXLAN on a virtual network interface and transmitted over UDP through port 8472 on the host network.

Prerequisites

An ACK managed Pro cluster that runs Kubernetes 1.33 or later.
If your cluster uses the Flannel network plugin, its version must be v0.15.1.23 or later.
A hybrid cloud node pool requires ECS nodes in the cluster to deploy certain ACK management components. Use three or more ECS nodes to ensure high availability.

Procedure

Step 1: Establish cross-cloud network

Before you install the Terway Hybrid plugin, ensure that the network path between your cloud and on-premises environments is fully connected:

Use Express Connect to create a dedicated connection and establish connectivity between the host networks of your VPC and on-premises data center. Also, configure route entries in the VPC route table and the Express Connect Router.
If your on-premises data center needs to access Alibaba Cloud services, add a route for the Alibaba Cloud service CIDR block 100.64.0.0/10 to your data center's core switch. For more information, see Configure a private connection over an Express Connect circuit.
If resources in the cloud need to access pods in your on-premises data center, the core switch must support BGP. For more information, see Step 3: Expose pods to external networks.

Step 2: Install the plugin

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Components and Add-ons.

Install the terway-hybrid-controlplane control plane component.

Parameter	Description
Hybrid Cloud Pod CIDR Block	The pod CIDR block for the hybrid cloud node pool. It cannot overlap with the existing service CIDR block, pod CIDR block, cloud node CIDR block, or on-premises node CIDR block.
Per-node pod CIDR block mask size	The subnet mask size for the pod CIDR block that is allocated to each node in the hybrid cloud node pool. For example, if you set this to 24, each node is allocated a /24 CIDR block, such as `xxx.xxx.xxx.0/24`, which provides 256 pod IP addresses.
SNAT	Specifies whether to translate the source IP address of pods in the hybrid cloud node pool to the node's IP address when they access external services. If you disable this option, ensure that external network devices can resolve pod IP addresses. For more information, see Step 3: Expose pods to external networks.

Install the terway-hybrid data plane component. The plugin automatically installs on new nodes that you add to the hybrid cloud node pool.

Step 3: Expose pods to external networks

If external network devices, such as those in the VPC or outside the cluster, need to directly access pods, configure BGP to dynamically advertise pod routes to your switches.

Create BGP configurations

Terway Hybrid defines the BGPClusterConfig CRD, which stores the BGP configuration and associates it with a hybrid cloud node pool by using nodeSelector.

Terway Hybrid does not create this resource by default. Create a BGPClusterConfig for each hybrid cloud node pool that requires BGP.

Create a file named bgpclusterconfig.yaml with the following code.

apiVersion: network.alibabacloud.com/v1beta1
kind: BGPClusterConfig
metadata:
  name: bgp
spec:
  localASN: 65010
  nodeSelector:
    matchLabels:
      alibabacloud.com/nodepool-id: "np-xxx"
  bgpSpeakers:
  - name: hybrid-node-1
    peers:
    - name: switch-1
      peerASN: 65001
      peerAddress: "10.10.0.1"
  - name: hybrid-node-2
    peers:
    - name: switch-1
      peerASN: 65001
      peerAddress: "10.10.0.1"
      #Optional
      authPassword:
        secretKeyRef:
          name: bgp-secret
          key: password
   
---
#Optional
apiVersion: v1
kind: Secret
metadata:
  name: bgp-secret
  namespace: kube-system
type: Opaque
data:
  password: bXxXXXxXXXXXx==  # Base64-encoded value.

Parameter	Required	Description
`metadata.name`	Yes	The name of the `BGPClusterConfig` in the cluster.
`spec.localASN`	Yes	The autonomous system number (ASN) of the BGP speaker's network. Use a private ASN from 64512 to 65535.
`spec.nodeSelector`	Yes	Specifies the nodes to which this BGP configuration applies. Use a label to select all nodes in a hybrid cloud node pool.
`spec.bgpSpeakers`	Yes	Specifies which of the associated nodes act as BGP speakers. These speakers advertise pod routes. Select two or more nodes to prevent a single point of failure.
`spec.bgpSpeakers.name`	Yes	The name of the BGP Speaker node must match the node name selected by `spec.nodeSelector`.
`spec.bgpSpeakers.peers`	Yes	A list of devices that establish a peering session with this BGP speaker. These devices are typically access-layer switches.
`spec.bgpSpeakers.peers.name`	Yes	The name of the BGP peer device.
`spec.bgpSpeakers.peers.peerASN`	Yes	The autonomous system number (ASN) of the BGP peer device.
`spec.bgpSpeakers.peers.peerAddress`	Yes	The IP address of the BGP peer device.
`spec.bgpSpeakers.peers.authPassword`	No	The authentication password for the BGP session is obtained by referencing a Secret resource. The Secret must be in the `kube-system` namespace.

Apply the BGPClusterConfig resource.
```
kubectl apply -f bgpclusterconfig.yaml
```

Configure external network devices

Enable the BGP service on the network devices in your on-premises data center and configure the nodes selected in Create BGP configurations as BGP peers. Then, on the VPC route table, Express Connect Router, and on-premises core switch, add routes that point to the on-premises pod CIDR block. After the configuration is complete, verify that BGP peering sessions are established.

Terway Hybrid enables BGP graceful restart by default with a duration of 600 seconds. Configure BGP graceful restart on your switches accordingly.