Phased release capability

Phased releases limit the blast radius of deployment failures by exposing changes to progressively larger scopes before full rollout.

A phased release is a progressive exposure model: instead of deploying changes to the entire production environment at once, you expose them incrementally—first to a controlled subset, then to broader scopes—so that failures surface early and affect as few users as possible. Two approaches support this model. A staging environment validates changes against a mirror of production traffic before they reach production. Batch deployment takes changes directly to production but limits scope at each step, letting you verify stability before proceeding.

Staging environment

A staging environment routes a controlled slice of production traffic through a fully isolated copy of the production stack. Because all internal calls form a closed loop within this environment, you can observe behavior without affecting real users.

The following figure shows how to build a staging environment.

Direct all internal network traffic and 1% of production traffic to the staging environment. Validate changes there before promoting them to production.

Batch deployment

Batch deployment takes changes to production in waves, giving you time to observe each wave before proceeding to the next. Three batching methods are available, applied at different levels of scope:

• Intra-cluster batching: Deploy in successive waves within a single cluster.

• Inter-cluster serial deployment: Deploy to clusters one at a time in sequence.

• Inter-cluster distribution: Distribute the deployment across clusters in parallel.

In these methods, a cluster is a logical group that can be further divided—examples include units, regions, data centers, zones, Virtual Private Clouds (VPCs), clusters, groups, and custom logical areas.

Requirements

Every phased release to a production environment must satisfy the following four requirements.

• Batching support: Use at least one of the three batching methods. Deploy in at least two batches—start with the smallest possible scope (ideally a single unit) to validate before expanding. If a phased release is not feasible, escalate the change for a higher level of approval.

• Controllable intervals: Control the time interval between batch deployments. For high-risk changes, the total phased release must run for at least one hour. For core system releases to production, the total observation period must be at least 30 to 60 minutes, with the first batch monitored for at least 20 minutes before proceeding. Set custom intervals for subsequent batches based on observed stability.

• Observability and validation: Monitor and validate the system after each batch. Do not proceed to the next batch until the current one is confirmed stable. Acceptable validation methods include: recording at least one core health metric (such as a business monitoring item or log file name), recording the personnel who perform a second confirmation, or using automated observation. Confirm deployment success by validating online files or using equivalent methods.

• Rollback capability: Be able to roll back a single batch or the entire release. All rollbacks must be recorded and traceable.