Stream Computing Architecture and High Availability
Architecture Overview
Stream computing adopts a two-tier architecture that separates the control plane from the data plane. The control plane handles job parsing, scheduling, and lifecycle management; the data plane performs actual data processing and state holding. The two are decoupled through an external metadata HA store. This separation makes management swappable, compute replaceable, and state replayable—together forming the HA closed loop of stream jobs. Both management and compute components support cross-zone disaster recovery.
The overall data flow: the control plane reads election and job information from the metadata HA store, dispatches compute tasks to the data plane, and collects runtime status. Multiple compute components on the data plane process data in parallel and periodically persist consistent snapshots to reliable state storage. When any component fails, the system recovers using the external metadata and the last successful snapshot.
Core Components
Management Component (Control Plane)
The management component handles SQL parsing and tuning, job lifecycle, resource orchestration, fault detection, recovery decisions, and snapshot coordination. It is deployed as multiple instances in an active-standby configuration: only one primary serves at a time; the rest are hot standbys. When the primary fails, distributed coordination performs leader election and automatically switches over, with the new primary taking over job management. Critical information—job definitions, recovery offsets—is not bound to any single management process's memory but persisted to the external HA store, avoiding permanent job loss from a single management failure.
Compute Component (Data Plane)
The compute component performs actual data processing and local state holding. It runs as a distributed multi-node cluster and scales elastically with job parallelism. A single node failure is isolated to the affected tasks; the system triggers task rescheduling and state recovery. The management component probes continuously via heartbeat; upon detecting an unreachable node, it reschedules that node's tasks to healthy nodes. Combined with consistent snapshots, recovery resumes from the most recent successful offset rather than replaying all history (requires snapshots to be enabled).
Metadata HA Storage / Distributed Coordination
This component persists job metadata, primary election information, and the critical pointers needed for recovery (such as the location of the latest completed consistent snapshot). Decoupled from compute nodes, it ensures the system can continue recovering jobs from unified metadata after a management active-standby switchover.
Reliable State Storage (Consistent Snapshots)
Once a job enables periodic consistent snapshots, distributed state is aligned to a recoverable boundary and persisted to reliable storage. This storage is the data foundation for fault recovery, ensuring state is replayable.
High Availability Mechanisms
Stream computing builds HA across three layers: management, compute, and state.
Management HA: active-standby election + externalized metadata. The management component runs multiple replicas with leader election via distributed coordination; job definitions and recovery offsets are written to the external HA store. If the primary dies, the standby takes over immediately and continues managing jobs from the external metadata—eliminating the management single point of risk.
Compute HA: node replacement + task rescheduling. The compute component runs as a multi-node cluster; a single node failure is isolated without affecting the rest. The management component probes continuously and reschedules a lost node's tasks to healthy nodes. Combined with consistent snapshots, processing resumes from the most recent successful offset, minimizing recomputation.
State consistency and exactly-once semantics: periodic consistent snapshots. A job can enable periodic consistent snapshots, aligning distributed state to a recoverable boundary and persisting it to reliable storage. After any management or compute failure, the system loads the last successful snapshot, aligns input offsets, and continues—supporting end-to-end consistency (the exact semantics depend on the connector and configuration). During public preview, manual recovery points (read-only, operationally rollback-capable) are also available for version changes or manual intervention.
Application-Level Fault Domain Isolation
Each stream application owns an independent management instance and compute resource view, achieving fault domain isolation between applications. Within a single application, "management active-standby + multiple compute nodes + external snapshot storage" forms that application's complete HA closed loop. When an application exits or fails, its runtime resources are reclaimed with it—avoiding the fault amplification of "one failure brings down everything" in large shared clusters, with better isolation.