Flink state compatibility-Realtime Compute for Apache Flink(Flink)-阿里云帮助中心

Realtime Compute for Apache Flink offers a state compatibility check and a state data migration feature. This topic explains the compatibility check results and compares the migration efficiency and job performance of the RocksDB and Gemini state backends.

Background

A Flink job uses state data, including checkpoints and savepoints, to store its intermediate computation results. As development and business requirements evolve, you often need to modify your SQL jobs. However, if you modify and restart an SQL job from its previous state, state incompatibility issues can occur.

Starting from engine version vvr-4.0.11-flink-1.13, Realtime Compute for Apache Flink offers a state compatibility check and a state data migration feature. These features help you reuse existing state data and update your SQL jobs quickly. After deploying a modified job, if you start it from a previous state, the system assesses its compatibility. For details about the results, see Compatibility.

For the new job to use the selected state data, the data must be migrated. Realtime Compute for Apache Flink supports two state backends, RocksDB and Gemini, which differ in migration efficiency and job performance. For more information, see State data migration.

Compatibility

On the Start Job page, when you select Resume Mode, the Flink system automatically detects changes to the SQL job, such as changes in the SQL code, Flink runtime parameters, or engine version. If changes are detected, we recommend clicking Click to detect next to State Compatibility to run a state compatibility check, then proceed based on the results.

Important

If you modify an SQL job, you must run a state compatibility check before starting it in Resume Mode to ensure a successful recovery.

The following list describes the compatibility results and our recommendations:

Fully compatible

The job's changes are fully compatible with the latest state data. This means the job will produce the same results as if it had processed the full data history. We recommend starting the job.
Partially compatible

The changes in the current job are partially compatible with the latest state data. This means that although results for compatible columns are consistent with the results from processing the full data history, other columns lack corresponding state data, which can lead to inconsistencies. To ensure full compatibility, we recommend that you start the job from a different state or perform a stateless start.
Incompatible

Warning
Starting the job from this state will likely cause it to fail or produce unexpected results. We strongly recommend using a different state or performing a stateless start.
Compatibility unknown

Warning
Starting the job from this state may cause it to fail or produce unexpected results. Proceed with caution.

State data migration

RocksDB and Gemini differ in migration efficiency and job performance:

RocksDB

RocksDB performs a full migration of all state data when the job starts. While the job is in the RUNNING state, operators requiring state data migration remain in the INITIALIZING state and do not process data. Once an operator's migration is complete, the operator transitions to RUNNING and begins to process data normally. In the Flink Web UI, the job has been running for 3m 8s, the Source operator's backpressure is at 100%, and the GroupAggregate operator shows 0 for both Bytes Received and Records Received.

Note
In this scenario, because an aggregate function was modified, RocksDB performs a full state migration at startup. The GroupAggregate operator remains in the INITIALIZING state and cannot process data until the migration is complete.
Gemini

Unlike RocksDB, which performs a full migration at startup, Gemini migrates state data on demand as the job runs. State data is migrated only upon first access. After startup, the job enters the RUNNING state, and its operators quickly transition from INITIALIZING to RUNNING and begin processing data. As the migration progresses, the transactions per second (TPS) gradually return to a normal level, indicating the migration is complete. Compared to RocksDB, Gemini typically has a shorter overall migration time. The job state_migration_q3_1_new_gemini is in the RUNNING state. After running for 1m 28s, the downstream GroupAggregate operator has received 20.4 MB (147,337 records), its Busy metric is at 100%, and the upstream Source operator's backpressure is at 99%.

Note
With the same SQL job and state data, Gemini begins processing data much faster than RocksDB because it transitions more quickly from the INITIALIZING to the RUNNING state.