SQL compatibility check-Realtime Compute for Apache Flink(Flink)-阿里云帮助中心

Realtime Compute for Apache Flink offers a state compatibility check and a state data migration feature. This topic explains the compatibility check results and compares the migration efficiency and job performance of the RocksDB and Gemini state backends.

Background

A Flink job uses state data, including checkpoints and savepoints, to store its intermediate computation results. Flink SQL now supports a wide range of stream processing scenarios. To meet development iteration and business growth requirements, SQL jobs change frequently. However, if you modify an SQL job and restart it from the original state data, state incompatibility issues can occur.

Starting from engine version vvr-4.0.11-flink-1.13, Realtime Compute for Apache Flink offers a state compatibility check and a state data migration feature. These features help you reuse existing state data and update your SQL jobs quickly. After deploying a modified job, if you start it from a previous state, the system assesses its compatibility. For details about the results, see Compatibility.

For the new job to use the selected state data, the data must be migrated. Realtime Compute for Apache Flink supports two state backends, RocksDB and Gemini, which differ in migration efficiency and job performance. For more information, see State data migration.

Compatibility

On the Start Job page, when you select Resume Mode, the Flink system automatically detects changes to the SQL job, such as changes in the SQL code, Flink runtime parameters, or engine version. If changes are detected, we recommend clicking Click to detect next to State Compatibility to run a state compatibility check, then proceed based on the results.

Important

When an SQL job has changes, you must run a state compatibility check before you select Resume Mode. The job can recover from the state in a compatible way only when all required conditions are met.

The following list describes the compatibility results and our recommendations:

Fully compatible

The job's changes are fully compatible with the latest state data. This means the job will produce the same results as if it had processed the full data history. We recommend starting the job.
Partially compatible

The changes in the current job are partially compatible with the latest state data. This means that although results for compatible columns are consistent with the results from processing the full data history, other columns lack corresponding state data, which can lead to inconsistencies. To ensure full compatibility, we recommend that you start the job from a different state or perform a stateless start.
Incompatible

Warning
Starting the job from this state will likely cause it to fail or produce unexpected results. Proceed with caution. We recommend that you use a different state or perform a stateless start.
Whether the current state is compatible needs to be determined after a restart (compatibility unknown)

Warning
Starting the job from this state may cause it to fail or produce unexpected results. Proceed with caution.

State data migration

RocksDB and Gemini differ in migration efficiency and job performance:

RocksDB

RocksDB performs a full migration of all state data when the job starts. While the job is in the RUNNING state, operators requiring state data migration remain in the INITIALIZING state and do not process data. Once an operator's migration is complete, the operator transitions to RUNNING and begins to process data normally. In the Flink Web UI, the job has been running for 3m 8s, the Source operator's backpressure is at 100%, and the GroupAggregate operator shows 0 for both Bytes Received and Records Received.

Note
Because the aggregate function was modified, RocksDB performs a full state migration when the job starts. The GroupAggregate operator is in the INITIALIZING state and cannot process data at this point.
Gemini

Unlike RocksDB, which performs a full migration at startup, Gemini migrates state data on demand as the job runs. State data is migrated only upon first access. After startup, the job enters the RUNNING state, and its operators quickly transition from INITIALIZING to RUNNING and begin processing data. As the migration progresses, the transactions per second (TPS) gradually return to a normal level, indicating the migration is complete. Compared to RocksDB, Gemini has a shorter overall migration time. The job state_migration_q3_1_new_gemini is in the RUNNING state. After running for 1m 28s, the downstream GroupAggregate operator has received 20.4 MB (147,337 records), its Busy metric is at 100%, and the upstream Source operator's backpressure is at 99%.

Note
When Gemini and RocksDB use exactly the same SQL and state data, Gemini quickly transitions from the INITIALIZING state (loading state data) to the RUNNING state and starts processing data.