Flink is one of the most active stream processing engines today, with relatively frequent version updates. Upgrading your job to a newer version enables you to use new features and capabilities. This topic explains how to upgrade the engine version for Realtime Compute for Apache Flink jobs.
Important notes
Before releasing a new engine version, Realtime Compute for Apache Flink performs extensive compatibility testing. In general, upgrades between minor versions within the same major version are compatible. Upgrades across major versions are not guaranteed to be compatible. For details about engine versions and Savepoint compatibility between Flink versions, see Engine version numbers and their meanings and the Compatibility Table.
When switching job versions, note the following:
-
State is compatible during minor version upgrades. You can use system checkpoints or job snapshots created before the version switch. For example, switching from vvr-4.0.15-flink-1.13 to vvr-4.0.18-flink-1.13.
-
State is not compatible during major version switches. You must restart the job in a stateless manner. For example, switching from vvr-4.0.15-flink-1.13 to vvr-6.0.2-flink-1.15.
-
The Flink-related dependency versions in your SQL or DataStream job must match the Flink version selected for the job.
-
Starting with Flink 1.13.0, the community adopted BlinkPlanner—contributed by Alibaba—as the default SQL Planner. This introduces some differences compared to earlier versions. For details, see the Apache Flink 1.13.0 release announcement. Therefore, when migrating from a community Flink version earlier than 1.13.0 to VVR 4.0 (based on Flink 1.13.0) or later in Realtime Compute for Apache Flink, you may encounter syntax and API incompatibilities.
Upgrade procedure
Step 1: Back up the job and deploy a new job
To ensure job stability, clone the existing job to create an identical new job, then upgrade the engine version on the new job.
SQL
-
Log on to the Realtime Compute console. Click Console in the Actions column for your target workspace.
-
Switch the new job to the new engine version.
When using a new version, choose a STABLE or RECOMMENDED version. These versions fix known bugs and issues from other versions, offer the latest features, and provide stronger stability guarantees.
-
Click More Settings on the right side of the new job editor. Switch the Engine Version, then click Deploy in the upper-right corner.
-
On the page, click the target job name. On the Deployment Details tab, under Basic Configuration, verify that the deployed job version has switched successfully.
Check the Engine Version field to confirm it shows the target version (for example,
vvr-8.0.9-flink-1.17).
-
DataStream
-
Click Deploy.
Step 2: Back up the job state
On the page, click the target job. On the State Set Management tab in the job details, view the job state sets. For more information, see View state generation overview.
-
If your job performs stateful computation, consider whether the state can be reused during the upgrade.
Before upgrading, manually create a job snapshot so you can quickly roll back if issues occur. For instructions, see Manually create a job snapshot.
Important-
In general, upgrades between minor versions within the same major version are compatible. Upgrades across major versions are not guaranteed to be compatible. If compatible, the new engine version can read snapshots generated by the old version—but not necessarily vice versa.
-
Starting with VVR 6.x, the system supports two snapshot formats: native and standard. The native format generates faster, while the standard format offers better compatibility. If upgrading to a higher minor version within the same major version, use the native format. If upgrading to a higher major version, use the standard format.
-
-
If your job is stateless, proceed to the next step.
(Optional) Step 3: Stop the original job
On the page, click Stop in the Actions column for the original job. For details, see Stop a job.
If your downstream writes are idempotent or your business can tolerate duplicate data, consider running both the old and new jobs simultaneously.
Step 4: Start the new job
-
If your job is stateful: Under Stateful Start, select Restore from Specified State and choose the snapshot created from the original job.
-
If your job is stateless, select Stateless to start the new job.
For detailed startup steps, see Start a job.
Step 5: Monitor production behavior and delete the original job
Typically, if the upgraded job starts successfully and completes its first system checkpoint, the upgrade is preliminarily successful. However, verify the correctness of your business data to ensure full upgrade integrity.
After confirming the job runs normally and data is correct, manually delete the pre-upgrade job snapshot and job. For instructions on deleting snapshots, see Manually delete a specified job snapshot.
Rollback after upgrade failure
If the job fails to start or produces incorrect data after the upgrade, immediately stop the job, switch back to the pre-upgrade engine version, and restore operations using the snapshot saved before the upgrade. Also, submit a ticket to report the issue.
If you cannot monitor the entire upgrade process, configure job failure alerts to receive immediate notifications in case of anomalies and prevent business impact. For instructions, see Configure Monitoring.