Hybrid cloud node pools let you connect on-premises data center servers to an ACK cluster over a leased line for containerization. Before nodes can join the cluster, you must configure network elements on both the on-premises side and the cloud side. This topic describes the required configuration for each network component.
If your network policy does not allow the entire 100.0.0.0/8 CIDR block for Alibaba Cloud services, use the per-product CIDR blocks listed in the Alibaba Cloud product CIDR block reference section instead.
Network architecture
The typical topology connecting an on-premises data center to an Alibaba Cloud VPC over a leased line is shown in the following description and figure.
-
On-premises data center: Servers and switches form Layer 2 local area networks (LANs). Multiple Layer 2 LANs combine into a Layer 3 network domain, with a private CIDR block of
10.0.0.0/8. -
Cloud VPC: A virtual private cloud (VPC) uses a virtualized network. Compute instances such as Elastic Compute Service (ECS) and Elastic Container Instance (ECI) connect directly to vSwitches. Alibaba Cloud services are accessible via the
100.0.0.0/8CIDR block, which is reachable from the VPC through vSwitches by default. -
Leased line: Alibaba Cloud provides Express Connect access points in multiple regions. Connect your data center's core switch to a Virtual Border Router (VBR) at the nearest Express Connect access point using a network firewall. Then connect the VBR to the cloud VPC using an Express Connect Router (ECR) or a Cloud Enterprise Network (CEN).
Configuration example
The following sections describe what to configure on each network component — both in your data center and in the cloud — to connect hybrid cloud nodes.
This example illustrates the configuration logic. Adapt it to your specific environment.
The example uses these parameters:
| Parameter | Value |
|---|---|
| Cloud VPC CIDR block | 192.168.0.0/16 |
| On-premises data center CIDR block | 10.0.0.0/8 |
| Alibaba Cloud service CIDR block | 100.0.0.0/8 |
| Hybrid cloud node container CIDR block | 172.16.0.0/12 |
| Cluster region | China (Hangzhou), cn-hangzhou |
The Layer 3 network configuration for the data center core switch and its subnets is assumed to be complete.
IDC network configuration
For hybrid cloud nodes to function correctly, your data center network must be able to reach these Alibaba Cloud services:
-
Object Storage Service (OSS)
-
Container Registry (ACR)
-
Cloud Assistant
The following sections describe how to configure each on-premises network component.
IDC core switch
| Configuration item | Description | Notes |
|---|---|---|
| Uplink route (data center to cloud) | Add routes pointing the cloud VPC CIDR block (192.168.0.0/16) and the Alibaba Cloud service CIDR block (100.0.0.0/8) to the Express Connect VBR. | If you use the Terway Hybrid plugin in Underlay mode, also configure Border Gateway Protocol (BGP) on the core switch and other switches to advertise container routes. For details, see Use the Terway Hybrid network plugin. |
| Downlink route (cloud to data center) | Accept traffic from the Express Connect VBR destined for data center servers. This example assumes the server network configuration is already complete on the core switch. | — |
IDC network firewall
| Configuration item | Description |
|---|---|
| Outbound (data center to cloud) | Allow traffic to the cloud VPC CIDR block (192.168.0.0/16) and the Alibaba Cloud service CIDR block (100.0.0.0/8). |
| Inbound (cloud to data center) | Allow traffic from the cloud VPC CIDR block (192.168.0.0/16) and the Alibaba Cloud service CIDR block (100.0.0.0/8). Open TCP 10250 and 10255 for kubelet, and TCP 9100 and 9445 for node-exporter. |
Node firewall rules
The following table lists all ports that must be open on hybrid cloud nodes to allow inbound access from the cloud. All source addresses refer to the VPC CIDR block.
| Protocol | Port | Source | Purpose |
|---|---|---|---|
| TCP | 10250, 10255 | VPC CIDR block | API Server and MetricsServer access kubelet for operations and metrics collection. |
| TCP | 9100, 9445 | VPC CIDR block | Prometheus accesses node-exporter for monitoring data. |
Outbound access from hybrid cloud nodes
The following table lists the endpoints and ports that hybrid cloud nodes must reach over the leased line. Replace {region} with the region ID of your cluster — for example, cn-hangzhou for China (Hangzhou). For region IDs, see the Alibaba Cloud product CIDR block reference section.
| Access target | Endpoint (leased line mode) | Port | Purpose |
|---|---|---|---|
| OSS — node component packages | aliacs-k8s-{region}.oss-{region}-internal.aliyuncs.com |
TCP 443 | Download installation packages for kubelet, CNI, container runtime, and auton-hub. |
| API Server internal endpoint | View on the Basic Information tab of the cluster. | TCP 6443 | Communicate with kube-apiserver. |
| System component image registry |
|
TCP 443 | Pull system component images. |
| Cloud Assistant |
|
TCP 443 | Cloud Assistant service endpoint and installation package. |
Cloud network configuration example
Virtual Border Router (VBR)
| Configuration item | Description | Notes |
|---|---|---|
| Uplink route (data center to cloud) | Configure the VBR to point the cloud VPC CIDR block (192.168.0.0/16) and the Alibaba Cloud service CIDR block (100.0.0.0/8) to the VPC. Use direct VPC connection, ECR, or CEN. For more information, see What is Express Connect?. | — |
| Downlink route (cloud to data center) | Configure the VBR to point the data center CIDR block (10.0.0.0/8) to the data center core switch through the leased line and firewall. | If you use the Terway Hybrid plugin in Underlay mode, also add the hybrid cloud node container CIDR block (172.16.0.0/12) to the VBR downlink route. |
ECR or CEN
| Configuration item | Description | Notes |
|---|---|---|
| Uplink route (data center to cloud) | In the ECR or CEN forwarding route table, point the cloud VPC CIDR block (192.168.0.0/16) and the Alibaba Cloud service CIDR block (100.0.0.0/8) to the VPC. | — |
| Downlink route (cloud to data center) | In the ECR or CEN forwarding route table, point the data center CIDR block (10.0.0.0/8) to the VBR. | If you use the Terway Hybrid plugin in Underlay mode, also add the hybrid cloud node container CIDR block (172.16.0.0/12) to the ECR or CEN downlink route. |
Cloud VPC route table
| Configuration item | Description | Notes |
|---|---|---|
| Uplink route (data center to cloud) | The VPC handles incoming traffic from the data center by default. No additional configuration is needed. | — |
| Downlink route (cloud to data center) | In the VPC route table, point the data center CIDR block (10.0.0.0/8) to the CEN or ECR. | If you use the Terway Hybrid plugin in Underlay mode, also add the hybrid cloud node container CIDR block (172.16.0.0/12) to the VPC route table downlink route. |
Cloud VPC security group
| Configuration item | Description |
|---|---|
| Outbound (cloud to data center) | Allow traffic to the data center CIDR block (10.0.0.0/8): TCP 10250 and 10255 for kubelet, and TCP 9100 and 9445 for node-exporter. |
| Inbound (data center to cloud) | Allow traffic from the data center CIDR block (10.0.0.0/8): TCP 6443 for the API Server. |
Alibaba Cloud product CIDR block reference
Use these tables when you cannot allow the full 100.0.0.0/8 CIDR block and need to allow traffic to specific CIDR blocks instead.
ACK component internal image addresses and CIDR blocks
| Region | Region ID | VPC network endpoint | Route CIDR block to add |
|---|---|---|---|
| China (Hangzhou) | cn-hangzhou | registry-cn-hangzhou-vpc.ack.aliyuncs.com |
100.103.9.188/32 100.103.7.181/32 |
| China (Shanghai) | cn-shanghai | registry-cn-shanghai-vpc.ack.aliyuncs.com |
100.103.94.158/32 100.103.7.57/32 100.100.80.231/32 |
| China (Fuzhou - Local Region - Shutting down) | cn-fuzhou | registry-cn-fuzhou-vpc.ack.aliyuncs.com |
100.100.0.43/32 100.100.0.28/32 |
| China (Qingdao) | cn-qingdao | registry-cn-qingdao-vpc.ack.aliyuncs.com |
100.100.0.172/32 100.100.0.207/32 |
| China (Beijing) | cn-beijing | registry-cn-beijing-vpc.ack.aliyuncs.com |
100.103.99.73/32 100.103.0.251/32 100.103.6.63/32 |
| China (Zhangjiakou) | cn-zhangjiakou | registry-cn-zhangjiakou-vpc.ack.aliyuncs.com |
100.100.1.179/32 100.100.80.152/32 |
| China (Hohhot) | cn-huhehaote | registry-cn-huhehaote-vpc.ack.aliyuncs.com |
100.100.0.194/32 100.100.80.55/32 |
| China (Ulanqab) | cn-wulanchabu | registry-cn-wulanchabu-vpc.ack.aliyuncs.com |
100.100.0.122/32 100.100.0.58/32 |
| China (Shenzhen) | cn-shenzhen | registry-cn-shenzhen-vpc.ack.aliyuncs.com |
100.103.96.139/32 100.103.6.153/32 100.103.26.52/32 |
| China (Heyuan) | cn-heyuan | registry-cn-heyuan-vpc.ack.aliyuncs.com |
100.100.0.150/32 100.100.0.193/32 |
| China (Guangzhou) | cn-guangzhou | registry-cn-guangzhou-vpc.ack.aliyuncs.com |
100.100.0.101/32 100.100.0.21/32 |
| China (Chengdu) | cn-chengdu | registry-cn-chengdu-vpc.ack.aliyuncs.com |
100.100.0.48/32 100.100.0.64/32 |
| Zhengzhou (Unicom Joint Venture) | cn-zhengzhou-jva | registry-cn-zhengzhou-jva-vpc.ack.aliyuncs.com |
100.100.0.111/32 100.100.0.84/32 |
| China (Hong Kong) | cn-hongkong | registry-cn-hongkong-vpc.ack.aliyuncs.com |
100.103.85.19/32 100.100.80.157/32 |
| US (Silicon Valley) | us-west-1 | registry-us-west-1-vpc.ack.aliyuncs.com |
100.103.13.55/32 100.100.80.93/32 |
| US (Virginia) | us-east-1 | registry-us-east-1-vpc.ack.aliyuncs.com |
100.103.12.19/32 100.100.80.11/32 |
| Japan (Tokyo) | ap-northeast-1 | registry-ap-northeast-1-vpc.ack.aliyuncs.com |
100.100.0.167/32 100.100.80.198/32 |
| South Korea (Seoul) | ap-northeast-2 | registry-ap-northeast-2-vpc.ack.aliyuncs.com |
100.100.0.71/32 100.100.0.33/32 |
| Singapore | ap-southeast-1 | registry-ap-southeast-1-vpc.ack.aliyuncs.com |
100.103.103.254/32 100.100.80.136/32 |
| Malaysia (Kuala Lumpur) | ap-southeast-3 | registry-ap-southeast-3-vpc.ack.aliyuncs.com |
100.100.0.17/32 100.100.80.137/32 |
| Indonesia (Jakarta) | ap-southeast-5 | registry-ap-southeast-5-vpc.ack.aliyuncs.com |
100.100.0.226/32 100.100.80.200/32 |
| Philippines (Manila) | ap-southeast-6 | registry-ap-southeast-6-vpc.ack.aliyuncs.com |
100.100.0.75/32 100.100.0.24/32 |
| Thailand (Bangkok) | ap-southeast-7 | registry-ap-southeast-7-vpc.ack.aliyuncs.com |
100.100.0.62/32 100.100.0.34/32 |
| Germany (Frankfurt) | eu-central-1 | registry-eu-central-1-vpc.ack.aliyuncs.com |
100.100.0.92/32 100.100.80.155/32 |
| UK (London) | eu-west-1 | registry-eu-west-1-vpc.ack.aliyuncs.com |
100.100.0.175/32 100.100.0.18/32 |
| Riyadh | me-central-1 | registry-me-central-1-vpc.ack.aliyuncs.com |
100.100.0.109/32 100.100.0.18/32 |
| China (Shanghai) Finance Cloud | cn-shanghai-finance-1 | registry-cn-shanghai-finance-1-vpc.ack.aliyuncs.com |
100.100.0.54/32 100.100.80.227/32 |
OSS internal domain names and VIP CIDR blocks
| Region | Region ID | OSS-specific region ID | VPC network endpoint | VIP CIDR block |
|---|---|---|---|---|
| China (Hangzhou) | cn-hangzhou | oss-cn-hangzhou | oss-cn-hangzhou-internal.aliyuncs.com |
100.118.28.0/24 100.114.102.0/24 100.98.170.0/24 100.118.31.0/24 |
| China (Shanghai) | cn-shanghai | oss-cn-shanghai | oss-cn-shanghai-internal.aliyuncs.com |
100.98.35.0/24 100.98.110.0/24 100.98.169.0/24 100.118.102.0/24 |
| China (Nanjing - Local Region - Shutting down) | cn-nanjing | oss-cn-nanjing | oss-cn-nanjing-internal.aliyuncs.com | 100.114.142.0/24 |
| China (Fuzhou - Local Region - Shutting down) | cn-fuzhou | oss-cn-fuzhou | oss-cn-fuzhou-internal.aliyuncs.com | 100.115.21.0/24 |
| China (Wuhan - Local Region) | cn-wuhan-lr | oss-cn-wuhan-lr | oss-cn-wuhan-lr-internal.aliyuncs.com | 100.115.89.0/24 |
| China (Qingdao) | cn-qingdao | oss-cn-qingdao | oss-cn-qingdao-internal.aliyuncs.com |
100.115.173.0/24 100.99.113.0/24 100.99.114.0/24 100.99.115.0/24 |
| China (Beijing) | cn-beijing | oss-cn-beijing | oss-cn-beijing-internal.aliyuncs.com |
100.118.58.0/24 100.118.167.0/24 100.118.170.0/24 100.118.171.0/24 100.118.172.0/24 100.118.173.0/24 |
| China (Zhangjiakou) | cn-zhangjiakou | oss-cn-zhangjiakou | oss-cn-zhangjiakou-internal.aliyuncs.com |
100.118.90.0/24 100.98.159.0/24 100.114.0.0/24 100.114.1.0/24 |
| China (Hohhot) | cn-huhehaote | oss-cn-huhehaote | oss-cn-huhehaote-internal.aliyuncs.com |
100.118.195.0/24 100.99.110.0/24 100.99.111.0/24 100.99.112.0/24 |
| China (Ulanqab) | cn-wulanchabu | oss-cn-wulanchabu | oss-cn-wulanchabu-internal.aliyuncs.com |
100.114.11.0/24 100.114.12.0/24 100.114.100.0/24 100.118.214.0/24 |
| China (Shenzhen) | cn-shenzhen | oss-cn-shenzhen | oss-cn-shenzhen-internal.aliyuncs.com |
100.118.78.0/24 100.118.203.0/24 100.118.204.0/24 100.118.217.0/24 |
| China (Heyuan) | cn-heyuan | oss-cn-heyuan | oss-cn-heyuan-internal.aliyuncs.com |
100.98.83.0/24 100.118.174.0/24 |
| China (Guangzhou) | cn-guangzhou | oss-cn-guangzhou | oss-cn-guangzhou-internal.aliyuncs.com |
100.115.33.0/24 100.114.101.0/24 |
| China (Chengdu) | cn-chengdu | oss-cn-chengdu | oss-cn-chengdu-internal.aliyuncs.com |
100.115.155.0/24 100.99.107.0/24 100.99.108.0/24 100.99.109.0/24 |
| China (Hong Kong) | cn-hongkong | oss-cn-hongkong | oss-cn-hongkong-internal.aliyuncs.com |
100.115.61.0/24 100.99.103.0/24 100.99.104.0/24 100.99.106.0/24 |
| US (Silicon Valley) | us-west-1 | oss-us-west-1 | oss-us-west-1-internal.aliyuncs.com | 100.115.107.0/24 |
| US (Virginia) | us-east-1 | oss-us-east-1 | oss-us-east-1-internal.aliyuncs.com |
100.115.60.0/24 100.99.100.0/24 100.99.101.0/24 100.99.102.0/24 |
| Japan (Tokyo) | ap-northeast-1 | oss-ap-northeast-1 | oss-ap-northeast-1-internal.aliyuncs.com |
100.114.211.0/24 100.114.114.0/25 |
| South Korea (Seoul) | ap-northeast-2 | oss-ap-northeast-2 | oss-ap-northeast-2-internal.aliyuncs.com | 100.99.119.0/24 |
| Singapore | ap-southeast-1 | oss-ap-southeast-1 | oss-ap-southeast-1-internal.aliyuncs.com |
100.118.219.0/24 100.99.213.0/24 100.99.116.0/24 100.99.117.0/24 |
| Malaysia (Kuala Lumpur) | ap-southeast-3 | oss-ap-southeast-3 | oss-ap-southeast-3-internal.aliyuncs.com |
100.118.165.0/24 100.99.125.0/24 100.99.130.0/24 100.99.131.0/24 |
| Indonesia (Jakarta) | ap-southeast-5 | oss-ap-southeast-5 | oss-ap-southeast-5-internal.aliyuncs.com | 100.114.98.0/24 |
| Philippines (Manila) | ap-southeast-6 | oss-ap-southeast-6 | oss-ap-southeast-6-internal.aliyuncs.com | 100.115.16.0/24 |
| Thailand (Bangkok) | ap-southeast-7 | oss-ap-southeast-7 | oss-ap-southeast-7-internal.aliyuncs.com | 100.98.249.0/24 |
| Germany (Frankfurt) | eu-central-1 | oss-eu-central-1 | oss-eu-central-1-internal.aliyuncs.com | 100.115.154.0/24 |
| UK (London) | eu-west-1 | oss-eu-west-1 | oss-eu-west-1-internal.aliyuncs.com | 100.114.114.128/25 |
| UAE (Dubai) | me-east-1 | oss-me-east-1 | oss-me-east-1-internal.aliyuncs.com | 100.99.235.0/24 |
| China (Hangzhou) Finance Cloud | cn-hangzhou-finance | oss-cn-hzjbp |
oss-cn-hzjbp-a-internal.aliyuncs.com oss-cn-hzjbp-b-internal.aliyuncs.com |
100.103.4.210/32 100.115.6.0/24 |
| China (Shanghai) Finance Cloud | cn-shanghai-finance-1 | oss-cn-shanghai-finance-1 | oss-cn-shanghai-finance-1-internal.aliyuncs.com |
100.115.105.0/24 100.100.36.8/32 |
| China (Beijing) Finance Cloud (Invitational Preview) | cn-beijing-finance-1 | oss-cn-beijing-finance-1 | oss-cn-beijing-finance-1-internal.aliyuncs.com | 100.112.52.0/24 |
| South China 1 Finance Cloud | cn-shenzhen-finance-1 | oss-cn-shenzhen-finance-1 | oss-cn-shenzhen-finance-1-internal.aliyuncs.com | 100.112.15.0/24 |
| Hangzhou Finance Cloud Public Network | cn-hangzhou-finance | oss-cn-hzfinance | oss-cn-hzfinance-internal.aliyuncs.com |
100.103.4.95/32 100.103.5.142/32 100.103.5.143/32 100.103.5.144/32 100.115.6.0/24 |
| Shanghai Finance Cloud Public Network | cn-shanghai-finance-1 | oss-cn-shanghai-finance-1-pub | oss-cn-shanghai-finance-1-pub-internal.aliyuncs.com |
100.100.36.24/32 100.100.36.8/32 |
| Shenzhen Finance Cloud Public Network | cn-shenzhen-finance-1 | oss-cn-szfinance | oss-cn-szfinance-internal.aliyuncs.com |
100.112.15.0/24 100.100.80.70/32 |
| Beijing Finance Cloud Public Network | cn-beijing-finance-1 | oss-cn-beijing-finance-1-pub | oss-cn-beijing-finance-1-pub-internal.aliyuncs.com | 100.112.52.0/24 |
Cloud Assistant service domain names and VIP CIDR blocks
| Region | Region ID | Domain name | IP address |
|---|---|---|---|
| China (Qingdao) | cn-qingdao | cn-qingdao.axt.aliyun.com |
100.100.15.4 100.100.183.1 |
| China (Beijing) | cn-beijing | cn-beijing.axt.aliyun.com | 100.100.18.120 |
| China (Zhangjiakou) | cn-zhangjiakou | cn-zhangjiakou.axt.aliyun.com |
100.100.99.23 100.100.202.194 |
| China (Hohhot) | cn-huhehaote | cn-huhehaote.axt.aliyun.com |
100.100.126.8 100.100.59.86 |
| China (Ulanqab) | cn-wulanchabu | cn-wulanchabu.axt.aliyun.com | 100.100.0.3 |
| China (Hangzhou) | cn-hangzhou | cn-hangzhou.axt.aliyun.com | 100.100.45.106 |
| China (Shanghai) | cn-shanghai | cn-shanghai.axt.aliyun.com |
100.100.36.108 100.100.159.7 |
| China (Nanjing - Local Region) | cn-nanjing | cn-nanjing.axt.aliyun.com | 100.100.0.1 |
| China (Fuzhou - Local Region) | cn-fuzhou | cn-fuzhou.axt.aliyun.com | 100.100.0.26 |
| China (Wuhan - Local Region) | cn-wuhan-lr | cn-wuhan-lr.axt.aliyun.com | 100.100.0.8 |
| China (Shenzhen) | cn-shenzhen | cn-shenzhen.axt.aliyun.com | 100.100.0.70 |
| China (Heyuan) | cn-heyuan | cn-heyuan.axt.aliyun.com | 100.100.0.5 |
| China (Guangzhou) | cn-guangzhou | cn-guangzhou.axt.aliyun.com | 100.100.0.4 |
| China (Chengdu) | cn-chengdu | cn-chengdu.axt.aliyun.com | 100.100.0.42 |
| China (Hong Kong) | cn-hongkong | cn-hongkong.axt.aliyun.com |
100.100.35.30 100.100.98.28 |
| Singapore | ap-southeast-1 | ap-southeast-1.axt.aliyun.com |
100.100.30.60 100.100.169.197 |
| Malaysia (Kuala Lumpur) | ap-southeast-3 | ap-southeast-3.axt.aliyun.com |
100.100.127.16 100.100.62.2 |
| Indonesia (Jakarta) | ap-southeast-5 | ap-southeast-5.axt.aliyun.com |
100.100.80.165 100.100.132.30 |
| Philippines (Manila) | ap-southeast-6 | ap-southeast-6.axt.aliyun.com | 100.100.0.15 |
| Thailand (Bangkok) | ap-southeast-7 | ap-southeast-7.axt.aliyun.com | 100.100.0.30 |
| Japan (Tokyo) | ap-northeast-1 | ap-northeast-1.axt.aliyun.com | 100.100.0.76 |
| South Korea (Seoul) | ap-northeast-2 | ap-northeast-2.axt.aliyun.com | 100.100.0.23 |
| US (Silicon Valley) | us-west-1 | us-west-1.axt.aliyun.com |
100.100.29.34 100.100.1.3 |
| US (Virginia) | us-east-1 | us-east-1.axt.aliyun.com |
100.100.152.140 100.100.147.87 |
| Germany (Frankfurt) | eu-central-1 | eu-central-1.axt.aliyun.com |
100.100.46.12 100.100.53.26 |
| UK (London) | eu-west-1 | eu-west-1.axt.aliyun.com | 100.100.0.20 |
| UAE (Dubai) | me-east-1 | me-east-1.axt.aliyun.com | 100.100.43.7 |
| China (Shanghai) Finance Cloud | cn-shanghai-finance-1 | cn-shanghai-finance-1.axt.aliyun.com | 100.100.0.46 |
| China (Beijing) Finance Cloud (Invitational Preview) | cn-beijing-finance-1 | cn-beijing-finance-1.axt.aliyun.com | 100.100.0.165 |
| China (Shenzhen) Finance Cloud | cn-shenzhen-finance-1 | cn-shenzhen-finance-1.axt.aliyun.com | 100.103.0.140 |
| China (Beijing) Alibaba Gov Cloud 1 | cn-north-2-gov-1 | cn-north-2-gov-1.axt.aliyun.com | 100.100.0.67 |