Agent Sandbox network planning and scaling

Background information

Sandboxes in Agent Sandbox run user-submitted code and are untrusted by default. To prevent sandboxes from accessing internal cluster components or other tenants, Agent Sandbox divides cluster workloads into three network partitions based on trust level (managed system components, unmanaged system components, and sandbox compute). We recommend using dedicated vSwitches and security groups to isolate sandboxes.

An enterprise security group supports up to 65,536 associated elastic network interfaces (ENIs). When the sandbox count in a security group nears this limit, or when a vSwitch runs out of IP addresses, create additional vSwitches and security groups for the new sandboxes.

Create a vSwitch

1. Log on to the or the ACS console. Click the target cluster name to open the Cluster Information page.

2. On the Basic Information tab, in the Network section, click the VPC name to open the VPC console.

3. In the VPC console, switch to the Resource Management tab, and Add a vSwitch. For instructions, see Create a VPC and a vSwitch.

   • Create at least one vSwitch in each zone to ensure high availability across zones.

   • Assign the CIDR block sequentially from existing ranges and avoid overlapping with occupied ranges.

In the VPC console, choose vSwitch from the left-side navigation pane to check the remaining Available IPv4 and avoid exhausting the address space.

Add a SNAT entry to the NAT gateway

A newly created vSwitch is not automatically added to the SNAT table of the NAT gateway. Without a SNAT entry, pods on the vSwitch cannot access the Internet (image pulls and external API calls will time out).

1. Open the Cluster Information page of the target cluster. On the Basic Information tab, in the Network section, click the NAT gateway name to open the NAT gateway console.

2. In the NAT gateway console, switch to the SNAT tab, Create SNAT Entry for the new vSwitch, and bind a NAT EIP. For instructions, see Create a SNAT entry.

Reuse or create a sandbox security group

1. On the Security Groups page, check whether an existing sandbox security group can be reused.

   Based on security group best practices, keep the IP utilization of a single security group below 80% (approximately 52,000 addresses).

2. On the Security Groups page, configure the allow rules one by one: sandbox egress rules, sandbox ingress rules, and destination-side ingress rules. For instructions, see Add a security group rule.

   The intra-group communication policy of an enterprise security group is isolated within the group. That is, instances in the same group are not connected by default. See Intra-group communication.

   Note

   Query the CIDR blocks of the components mentioned in the following tables:

   • Cluster-managed components

     • ACS/ACK: Open the Cluster Information page of the target cluster. On the Basic Information tab, in the Network section, view the Control Plane vSwitch information.

   • Cluster-unmanaged components

     • ACS: Open the Cluster Information page of the target cluster. On the Basic Information tab, in the Network section, view the Control Plane vSwitch information.

     • ACK: Open the Node Pools page of the target cluster. Query the vSwitch information of the corresponding node pool.

   Sandbox security group ingress rules

   Source	Protocol/Port	Purpose	Action
   Cluster-managed components CIDR block	TCP/9090, TCP/9100, TCP/10254	Prometheus / node-exporter / Ingress metrics	Allow
   	TCP/49983	ack-agent-sandbox-controller	Allow
   	ICMP	Health probes	Allow
   Cluster-unmanaged components CIDR block	TCP/49983, TCP/49999	sandbox-manager and sandbox-gateway→ sandbox pod (envd / jupyter)	Allow

   Sandbox security group egress rules

   Destination	Protocol/Port	Purpose	Action	Priority
   100.100.100.200/32	TCP/80, TCP/443	ECS metadata service	Deny	1
   Cluster-managed components CIDR block	TCP/6443	API Server	Allow	1
   	TCP/9082	Poseidon managed components	Allow	1
   	TCP/53, UDP/53	Private DNS (PrivateZone) / DNS upstream	Allow	1
   	TCP/49983	sandbox-controller service	Allow	1
   	TCP/8443	ack-agent-identity service	Allow	1
   Cluster-managed / unmanaged components CIDR block	TCP/53, UDP/53	Sandbox access to CoreDNS pods	Allow	1
   vSwitch CIDR block of ack-sandbox-manager	TCP/15008 TCP/15012	enhanced-traffic-policy network proxy gateway	Allow	1
   OSS, NAS, and other service addresses	ALL	ossfs and NAS service access	Allow	1
   10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16	ALL	Private networks (prevents sandbox probing of the intranet)	Deny	10
   0.0.0.0/0	ALL	Internet egress Open as needed.	Allow	20

   A smaller priority value indicates a higher priority. At the same priority, deny rules take precedence.

   Destination-side security group ingress rules

   After adding a vSwitch, the security groups of the target components (CoreDNS, API Server, and so on) that the sandbox pods access must include ingress rules that allow the CIDR block of the new vSwitch.

   Source	Protocol/Port	Purpose	Required
   New sandbox vSwitch CIDR block	TCP/53, UDP/53	Optional. Sandbox access to CoreDNS for domain resolution.	No
   	TCP/6443	ACS instance access to the cluster API Server	Yes
   	TCP/9082	ACS instance access to Poseidon managed components	No

Configure vSwitch and security group on pods

apiVersion: agents.kruise.io/v1alpha1
kind: SandboxSet
metadata:
  name: code-interpreter
  namespace: default
spec:
  template:
    metadata:
      annotations:
        # vSwitch IDs. Separate multiple IDs with commas.
        network.alibabacloud.com/vswitch-ids: "vsw-uf6h*********oy7n,vsw-uf6vx********9c"
        # Security group IDs. Separate multiple IDs with commas.
        network.alibabacloud.com/security-group-ids: "sg-uf6g**********i0dn"
 # ...

Adjust TrafficPolicy / NetworkPolicy

After scaling, check whether existing TrafficPolicy (apiVersion: network.alibabacloud.com/v1alpha1) rules cover the new CIDR block. For details, see Manage Agent network access using TrafficPolicy and Enhanced egress traffic management for Agent Sandbox.

Verify network connectivity

After scaling, use a sandbox pod or a temporary test pod to verify each network path.

curl https://httpbin.org/ip

# Query the domain of an internal cluster service.
getent hosts kubernetes.default.svc.cluster.local

Related documents

• Manage Agent network access using TrafficPolicy

• Enhanced egress traffic management for Agent Sandbox

• Configure an ACL for ALB Ingress to implement access control