Deploy and configure the Terway plugin-Container Service for Kubernetes(ACK)-阿里云帮助中心

Hybrid clusters use container network plugins on both on-premises data center nodes and cloud compute nodes. This topic explains how to deploy and configure the Terway network plugin on cloud nodes.

Prerequisites

Ensure that you have:

Configured the container network plugins for both cloud and on-premises nodes. See Configure container network plugins for cloud and on-premises nodes.
For Scenario 2: BGP network and Scenario 3: Host network, configured the following Terway network parameters when creating the registered cluster. See Create an ACK One registered cluster.
- Selected or cleared the IPvlan checkbox as needed.
- Configured the pod vSwitch.
- Configured the Service CIDR block.

Choose a scenario

Choose the scenario that matches your on-premises container network.

Scenario	On-premises network type	Cloud nodes requirement
Scenario 1: Overlay network	Flannel VXLAN, Calico IPIP, or Cilium VXLAN	No additional setup — cloud nodes can use the same mode
Scenario 2: BGP network	Border Gateway Protocol (BGP)	Must use Terway. Follow the install steps below.
Scenario 3: Host network	Host network	Must use Terway. Follow the install steps below.

Scenario 1: Overlay network

If your on-premises network uses an overlay, cloud nodes can use the same mode. Ensure cloud nodes can pull the images required by the network plugin DaemonSet.

Common overlay network modes include:

Flannel VXLAN
Calico IPIP
Cilium VXLAN

No additional installation is required for this scenario.

Scenario 2: BGP network

If your on-premises network uses BGP, cloud nodes must use Terway. To enable cloud-to-on-premises network communication, see Configure BGP on a Virtual Border Router (VBR).

Cloud nodes added by node pool scale-out get the alibabacloud.com/external=true label. By default, Terway is scheduled only to these nodes.

To prevent scheduling conflicts:

The on-premises CNI DaemonSet (for example, Calico in BGP route reflector mode) is not scheduled to cloud nodes.
The Terway DaemonSet is not scheduled to on-premises nodes.

Prevent the on-premises CNI from running on cloud nodes

Use nodeAffinity to prevent the Calico DaemonSet from being scheduled to nodes with the alibabacloud.com/external=true label. Apply this to any workload that must stay on-premises.

Warning

Patching the Calico DaemonSet restarts it on all affected nodes. Misconfigured affinity may disrupt pod networking on on-premises nodes. Verify the patch before applying.

Update the Calico DaemonSet:

cat <<EOF > calico-ds.patch
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: alibabacloud.com/external
                operator: NotIn
                values:
                - "true"
EOF
kubectl -n kube-system patch ds calico-node -p "$(cat calico-ds.patch)"

After this patch, the Calico DaemonSet (calico-node in kube-system) runs only on on-premises nodes. Terway continues to run only on cloud nodes.

After verifying the scheduling constraints, proceed to install and configure the Terway plugin.

Scenario 3: Host network

If your on-premises network uses host networking, ensure the Terway DaemonSet is not scheduled to on-premises nodes. By default, Terway runs only on cloud nodes with the alibabacloud.com/external=true label, so no extra configuration is needed.

Proceed to install and configure the Terway plugin.

Install and configure the Terway plugin

Follow these steps for Scenario 2 or Scenario 3. Each step offers two methods: the console or the onectl CLI.

Tip: Use onectl for automation and scripted deployments. Use the console for one-time setups.

Step 1: Configure RAM permissions

Terway needs Resource Access Management (RAM) permissions to manage elastic network interfaces (ENIs) on Elastic Compute Service (ECS) instances.

Configure in the console

Create a RAM user and attach the following custom policy. See Use RAM to grant access permissions to clusters and cloud resources.

{
    "Version": "1",
    "Statement": [
        {
            "Action": [
                "ecs:CreateNetworkInterface",
                "ecs:DescribeNetworkInterfaces",
                "ecs:AttachNetworkInterface",
                "ecs:DetachNetworkInterface",
                "ecs:DeleteNetworkInterface",
                "ecs:DescribeInstanceAttribute",
                "ecs:AssignPrivateIpAddresses",
                "ecs:UnassignPrivateIpAddresses",
                "ecs:DescribeInstances",
                "ecs:ModifyNetworkInterfaceAttribute"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "vpc:DescribeVSwitches"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        }
    ]
}

Log on to the Container Service Management Console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, choose Configurations > Secrets.

On the Secrets page, click Create from YAML. Enter the following YAML to create a secret named alibaba-addon-secret.

Terway uses the AccessKey pair in this secret to access cloud services. Skip if the secret already exists.

apiVersion: v1
kind: Secret
metadata:
  name: alibaba-addon-secret
  namespace: kube-system
type: Opaque
stringData:
  access-key-id: <AccessKey ID of the RAM user>
  access-key-secret: <AccessKey secret of the RAM user>

Configure using onectl

Install onectl on your on-premises machine. See Use onectl to manage registered clusters.

Grant RAM permissions to Terway:

onectl ram-user grant --addon terway-eniip

Expected output:

Ram policy ack-one-registered-cluster-policy-terway-eniip granted to ram user ack-one-user-ce313528c3 successfully.

Step 2: Install the Terway plugin

Install from the console

Log on to the Container Service Management Console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Add-ons.
On the Add-ons page, click the Network tab. In the terway-eniip section, click Install.

Install using onectl

Install Terway:

onectl addon install terway-eniip

Expected output:

Addon terway-eniip, version **** installed.

Next steps

To set up BGP connectivity between cloud and on-premises containers, see Configure BGP on a VBR.