One-click major engine version upgrade

Prerequisites

The cluster is in the Running state.

Usage notes

• You cannot cancel the one-click major engine version upgrade process after it starts.

  Note

  A version rollback is not supported after a one-click major engine version upgrade. If you need to be able to roll back to the previous version after the upgrade, you can perform the upgrade by cloning the cluster. For more information, see Upgrade the major engine version by cloning.

• To ensure a smooth upgrade, we recommend that you stop write operations from your application before you start the upgrade. Although the cluster automatically stops write operations during the process, failing to stop them on the application side beforehand may cause long delays or prevent synchronization from completing promptly. This can slow down the upgrade.

• To minimize the impact on your business, we strongly recommend that you perform the following validations before you upgrade:

  • Compatibility and performance validation: Verify any differences in features, syntax, and performance for write and query operations.

  • Upgrade duration validation: The instance restarts multiple times during the version upgrade. For some instances, a single restart may take a long time due to a large number of databases, tables, or partitions (parts).

    Important

    If the cluster uses Tiered Storage for Hot and Cold Data and you perform a clone validation, the one-click upgrade will take longer than the validation. This is because the clone operation does not include cold data, while the actual upgrade does.

  For information about how to perform the validation, see Validate version compatibility.

Impact on the cluster

The cluster restarts multiple times during the version upgrade. During these restarts, the cluster is unavailable for both read and write operations. We recommend that you perform the upgrade during off-peak hours.

Validate version compatibility

Because a one-click major engine version upgrade cannot be rolled back, we strongly recommend that you test the upgrade first. Confirm that the new version is fully compatible with the features of the source version and that the upgrade duration is acceptable before you proceed. Follow these steps to perform the validation:

1. Navigate to the Cluster Information page of the target instance. In the left-side navigation pane, click Backup and Restoration. After the instance is backed up, click Restore Instance.

2. Select Clone from real-time replica and set the target engine version to the version you want to upgrade to.

3. Create the cloned cluster.

4. Perform compatibility validation.

   1. Validate the compatibility of your business queries with the new version. For more information, see SQL compatibility validation.

   2. Perform regression testing for your business features.

Procedure

1. Log in to the ApsaraDB for ClickHouse console with your Alibaba Cloud account.

2. In the upper-left corner of the page, select the region where the target cluster is located.

3. In the left-side navigation pane, click Clusters of Community-compatible Edition.

4. Find the target cluster and click the cluster ID to go to the Cluster Information page.

5. In the Cluster Properties section, click Major Version Upgrade next to Version.

6. Configure the following parameters as prompted and click OK.

   Parameter	Description	Example
   Upgrade cluster kernel version to	The target cluster version. A version rollback is not supported after the upgrade. Currently, only version 23.8 is supported.	23.8 (LTS version)
   Version upgrade execution time	The time to perform the cluster upgrade. Important If you select to upgrade at a specified time or within a maintenance window, the cluster status changes from Running to Upgrading. Before the specified time or maintenance window arrives, the cluster can serve read and write requests normally, but you cannot perform maintenance operations such as instance configuration changes, scaling, or migration. Specified Time: Select a future point in time to perform the major version upgrade. Upgrade Within Maintenance Window: The upgrade is performed during the cluster's configured maintenance window. Update Now: Starts the upgrade immediately.	2024-05-29 14:46
   Perform clone validation	Select Clone Validation Performed or Skip Clone Verification (Not Recommended).	Clone validation performed

Prerequisites

The cluster is in the Running state.

Usage notes

• A one-click major engine version upgrade involves only a single cluster, and you cannot perform a version rollback after the upgrade is complete.

  Note

  A version rollback is not supported after a one-click major engine version upgrade. If you need to be able to roll back to the previous version after the upgrade, you can perform the upgrade by migrating data. For more information, see Upgrade the major engine version by data migration.

• When you upgrade a cluster, note the following points about its databases and tables:

  • For tables that use a MergeTree-family engine, historical data is migrated to the new cluster and automatically redistributed during the upgrade.

  • For tables that do not use a MergeTree-family engine, such as external tables or Log tables, only the table schemas are migrated. Data is not migrated.

  • For a materialized view, only the schema is migrated. Data is not migrated.

  • Kafka and RabbitMQ engine tables cannot be directly migrated through the console upgrade feature. You must record the DDL statements of these tables and delete them from the cluster before upgrading to prevent interference with the upgrade task. After the upgrade is complete, you must rebuild these tables. For detailed steps, see the procedure below.

  • The internal node IP addresses change after the upgrade. If your data writes and access depend on node IP addresses, you must obtain the new VPC CIDR block of the cluster.

• To minimize the impact on your business, we strongly recommend that you perform the following validations before you upgrade:

  • Compatibility and performance validation: Verify any differences in features, syntax, and performance for write and query operations.

  • Upgrade duration validation: The instance restarts multiple times during the version upgrade. For some instances, a single restart may take a long time due to a large number of databases, tables, or partitions (parts).

  For information about how to perform the validation, see Validate version compatibility.

Impact on the cluster

A Community Edition cluster is readable and writable throughout the upgrade process, except for the final 10 minutes of migration when the cluster becomes read-only. To check the remaining migration time, see View upgrade progress.

Validate version compatibility

Because a one-click major engine version upgrade cannot be rolled back, we strongly recommend that you test the upgrade first. Confirm that the new version is fully compatible with the features of the source version and that the upgrade duration is acceptable before you proceed. Follow these steps to perform the validation:

1. Purchase a new cluster to perform migration validation. For more information, see Upgrade the major engine version by data migration.

2. In the new cluster, perform SQL compatibility validation. For more information, see SQL compatibility validation.

3. Perform regression testing for your business features.

Step 1: Record and clean up Kafka/RabbitMQ engine tables

Kafka and RabbitMQ engine tables cannot be directly migrated through the console upgrade feature. You must record the DDL statements of these tables and delete them from the cluster before upgrading to prevent interference with the upgrade task.

1. Log on to the cluster and run the following statement to query the tables that need to be handled.

   /*
   create_table_query: table definition statement
   dependencies_database: database that depends on this table
   dependencies_table: table name that depends on this table
   Use dependencies_database and dependencies_table to identify materialized views that depend on Kafka/RabbitMQ tables
   */
   SELECT * FROM system.tables WHERE engine IN ('RabbitMQ', 'Kafka');

2. View the materialized view definition to check whether its target table is an implicit table.

   /*
   View the materialized view definition.
   If the target table of the materialized view is an implicit table, note the following:
   Deleting the materialized view will also delete the implicit table, resulting in data loss.
   Example: If CREATE MATERIALIZED VIEW [db.]table_name [TO[db.]name] does not specify the TO clause,
   the system automatically creates an implicit table in the format '.inner_id.<TABLE_UUID>' or '.inner.<TABLE>'
   */
   SELECT * FROM system.tables WHERE database='<DATABASE>' AND name = '<MATERIALIZED_VIEW_NAME>';

3. If a materialized view was created without the TO clause, the system automatically generates an implicit target table prefixed with .inner or .inner_id. These implicit tables are migrated to the new cluster nodes during upgrading, but due to internal naming mechanisms, directly recreating the original materialized view may cause naming conflicts. Therefore, you must first rename the implicit target tables to regular table names.

   -- Rename implicit target tables to regular table names (execute on all nodes)
   RENAME TABLE <database>.`.inner_id.<uuid>` TO <database>.<new_target_table_name> ON CLUSTER default;

4. Delete the materialized views and Kafka/RabbitMQ engine tables.

   Important

   You must delete the materialized views that reference the Kafka/RabbitMQ tables first, and then delete the Kafka/RabbitMQ engine tables. Otherwise, the upgrade operation fails.

   -- Delete materialized views first
   DROP TABLE <database>.<materialized_view_name> ON CLUSTER default;
   
   -- Then delete Kafka/RabbitMQ engine tables
   DROP TABLE <database>.<kafka_or_rabbitmq_table_name> ON CLUSTER default;

Step 2: Upgrade the cluster

1. Log in to the ApsaraDB for ClickHouse console with your Alibaba Cloud account.

2. In the upper-left corner of the page, select the region where the target cluster is located.

3. In the left-side navigation pane, click Clusters of Community-compatible Edition.

4. Find the target cluster and click the cluster ID to go to the Cluster Information page.

5. In the Cluster Properties section, click Major Version Upgrade next to Version.

6. Configure the following parameters as prompted and click OK.

   Parameter	Description	Example
   Upgrade cluster kernel version to	The target cluster version. A version rollback is not supported after the upgrade. Currently, only version 23.8 is supported.	23.8 (LTS version)
   Time of Stopping Data Writing	To ensure data consistency before and after the upgrade, the cluster automatically stops write operations during the last 10 minutes of the upgrade. The following rules apply to setting the write suspension time: To ensure a successful upgrade, we recommend that you set the write suspension time to at least 30 minutes. The upgrade must finish within 5 days. Therefore, the end date for the Time of Stopping Data Writing must be less than or equal to current date + 5 days. To minimize the impact of write suspension on your business, we recommend that you set the write suspension time to a period during your off-peak hours.	2025-03-20 10:08 - 2025-03-25 10:08
   Perform instance migration validation	Select Verification Performed or Skip instance migration validation (Not Recommended).	Instance migration validation performed

7. Next steps

   • If the source cluster contains Kafka or RabbitMQ engine tables, perform Steps 3 through 5 at the appropriate times during the upgrade. For details, see the sections below.

   • To view the upgrade progress, see View the upgrade progress.

   • If the specified write suspension time has passed but the cluster status is still Upgrading, you must modify the write suspension time to complete the migration.

   • If the upgrade affects your business and you want to stop it quickly, you can cancel the upgrade.

Step 3: Rebuild Kafka/RabbitMQ engine tables during data migration

When the upgrade task enters the Data Migration phase (after table schema migration is complete), use the previously saved DDL statements to rebuild the Kafka/RabbitMQ engine tables and their downstream materialized views. Once rebuilt, incremental data resumes flowing and is automatically synchronized to the new cluster nodes.

-- Rebuild Kafka/RabbitMQ engine tables
CREATE TABLE <database>.<kafka_or_rabbitmq_table_name> (...)
ENGINE = Kafka/RabbitMQ
SETTINGS ...;

-- Rebuild materialized view (using TO clause pointing to the renamed target table)
CREATE MATERIALIZED VIEW <database>.<materialized_view_name> TO <database>.<new_target_table_name>
AS SELECT ... FROM <database>.<kafka_or_rabbitmq_table_name>;

Step 4: Delete Kafka/RabbitMQ engine tables before write suspension

Shortly before the configured write suspension window, delete the Kafka/RabbitMQ engine tables and their downstream materialized views on the cluster to stop incremental data writes and ensure consistency of the final data synchronization.

-- Delete materialized views first
DROP TABLE <database>.<materialized_view_name> ON CLUSTER default;

-- Then delete Kafka/RabbitMQ engine tables
DROP TABLE <database>.<kafka_or_rabbitmq_table_name> ON CLUSTER default;

Step 5: Rebuild Kafka/RabbitMQ engine tables after upgrade

After the upgrade task is complete, use the previously saved DDL statements to rebuild the Kafka/RabbitMQ engine tables and their downstream materialized views on the upgraded cluster to restore the incremental data consumption pipeline.

-- Rebuild Kafka/RabbitMQ engine tables
CREATE TABLE <database>.<kafka_or_rabbitmq_table_name> (...)
ENGINE = Kafka/RabbitMQ
SETTINGS ...;

-- Rebuild materialized view (using TO clause pointing to the renamed target table)
CREATE MATERIALIZED VIEW <database>.<materialized_view_name> TO <database>.<new_target_table_name>
AS SELECT ... FROM <database>.<kafka_or_rabbitmq_table_name>;

View the upgrade progress

1. Log in to the ApsaraDB for ClickHouse console with your Alibaba Cloud account.

2. In the upper-left corner of the page, select the region where the target cluster is located.

3. In the left-side navigation pane, click Clusters of Community-compatible Edition.

4. Find the target cluster and click the cluster ID to go to the Cluster Information page.

5. In the Cluster Status section, click View Progress next to Status.

   The Modify Data Write-Stop Time Window dialog box appears, where you can view the upgrade progress. For example, you can view the progress of MergeTree schema migration, data migration, the estimated remaining time for data migration, and the progress of other schema migrations.

Modify the write suspension time

If the specified write suspension time has passed but the cluster status is still Upgrading, the data migration is not yet complete. You must adjust the write suspension time to ensure that the upgrade finishes successfully.

1. Log in to the ApsaraDB for ClickHouse console with your Alibaba Cloud account.

2. In the upper-left corner of the page, select the region where the target cluster is located.

3. In the left-side navigation pane, click Clusters of Community-compatible Edition.

4. Find the target cluster and click the cluster ID to go to the Cluster Information page.

5. In the Cluster Status section, click View Progress next to Status.

6. In the Modify Data Write-Stop Time Window dialog box that appears, modify the Time of Stopping Data Writing and click Confirm.

   Note

   The rules for setting the Time of Stopping Data Writing are the same as those for the Time of Stopping Data Writing parameter described in Upgrade the cluster.

Cancel the upgrade

If the upgrade affects your business and you want to stop it quickly, you can cancel the upgrade.

1. Log in to the ApsaraDB for ClickHouse console with your Alibaba Cloud account.

2. In the upper-left corner of the page, select the region where the target cluster is located.

3. In the left-side navigation pane, click Clusters of Community-compatible Edition.

4. Find the target cluster and click the cluster ID to go to the Cluster Information page.

5. In the Cluster Status section, click View Progress next to Status.

6. In the Modify Data Write-Stop Time Window dialog box that appears, click Cancel Upgrade.

   Note

   After you click Cancel Upgrade, the upgrade task does not stop immediately. The task stops completely after about 5 minutes.