Multi-Cluster Agent Sandbox

Single-cluster capacity control: Each member cluster is configured with a maximum number of Sandboxes that can run (Capacity). When the limit is exceeded, the scheduler automatically schedules new Sandboxes to other clusters that still have capacity, preventing a single cluster from being overloaded.

Utilization-balanced scheduling: By default, Sandboxes are evenly distributed based on the current Sandbox usage level of each cluster, to maximize multi-cluster resource utilization.

Warm pool awareness and affinity scheduling: The scheduler prefers clusters with sufficient prewarmed Sandboxes, ensuring that Sandboxes start as fast as possible and improving agent startup speed. This is highly valuable in popular reinforcement learning (RL) training scenarios.

Multi-cluster rate-limit scheduling: The rate at which Create Sandbox requests are sent to each member cluster can be individually controlled to avoid overloading the control plane of member clusters. Whenever possible, requests are scheduled to clusters that still have traffic headroom. This can be configured in the cluster scheduling configuration file sandbox-e2b-scheduler-config.

Persistent Volume (PV) creation follows Sandbox scheduling: If each Sandbox requires a dedicated PV, PVs can be created at the fleet level, and region-aware scheduling based on the region of the PV is supported. After the Global Sandbox Scheduler schedules a Sandbox, the PV is created in the target cluster as well. This reduces the cost of managing PVs across multiple clusters. Meanwhile, PVs in member clusters that are not used by any Sandbox are automatically garbage-collected (GC).

Multi-cluster failover: When a cluster becomes abnormal, new Sandbox creation requests can be scheduled to healthy clusters. Both manual and automatic switching are supported. Manual switching is particularly suitable for cluster O&M: it ensures that new Sandbox creation requests are not scheduled to the cluster under maintenance and do not affect normal access to existing Sandboxes on that cluster (such as Connect, hibernation and wake-up, and code execution).

Single-cluster capacity control: Each member cluster is configured with a maximum number of Sandboxes that can run (Capacity). When the limit is exceeded, the scheduler automatically schedules new Sandboxes to other clusters that still have capacity, preventing a single cluster from being overloaded.

Utilization-balanced scheduling: By default, Sandboxes are evenly distributed based on the current Sandbox usage level of each cluster, making full use of multi-cluster resources.

Cluster priority scheduling: Clusters can be divided into priority groups. Sandboxes are first scheduled to the specified clusters (groups), and when resources are insufficient, they are automatically downgraded to the clusters (groups) of the next priority level.

Warm pool awareness and affinity scheduling: The scheduler prefers clusters with sufficient prewarmed Sandboxes, ensuring that Sandboxes start as fast as possible and improving agent startup speed. This is highly valuable in popular RL training scenarios.

Multi-cluster failover: When a cluster becomes abnormal, new Sandbox creation requests can be scheduled to healthy clusters. Both manual and automatic switching are supported. Manual switching is particularly suitable for cluster O&M: it ensures that new Sandbox creation requests are neither scheduled to the cluster under maintenance nor affect normal access to existing Sandboxes on that cluster (such as Connect, hibernation and wake-up, and code execution).