PolarDB-X CDC release notes

Category

Description

Fix

Fixed a circular dependency issue during CDC BinlogX Relay Log cleanup.

Category

Description

Enhancement

Fixed exceptions that occurred during asynchronous DDL synchronization.

Fix

Fixed an issue where the binlog dump could hang after a CDC version upgrade and a binlog rebuild.

Category

Description

Fixed

Fixed an issue where certain invalid table creation statements caused Binlog to generate incorrect SQL.

Category

Description

Enhancements

Metadata persistence is now automatically enabled for low memory configurations.

Fixed an issue where a replica could not synchronize empty string values for non-NULL JSON fields.

Fixes

Fixed an issue where multiple CREATE DATABASE IF NOT EXISTS statements could interrupt a log node during data node scale-up or scaling.

Fixed an issue where a log node would occasionally report an error when retrieving the METADB address.

Category

Description

New features and enhancements

Upgraded CDC (log node) to version 5.4.21.

The CDC (log node) now supports the new DN query event variable Q_OPT_INDEX_FORMAT_HIPPO_ENABLED.

Introduced support for Binlog compression.

Improved the data synchronization performance of the columnar store index (CCI).

Optimized the recovery speed of the CDC (log node) when a storage node (DN) undergoes a specification change.

Fixes

Fixed an error that occurred when downstream data synchronization tools, such as Flink CDC, parsed Binlog events of the FORMAT_DESCRIPTION_EVENT type.

Fixed an error in the columnar store index (CCI) when it parsed DDL statements containing the comment character #.

Fixed an occasional error in the Binlog consumption pipeline that occurred during high-rate Binlog generation.

Fixed intermittent connection interruptions to downstream PolarDB-X instances caused by source-side filtering on the CDC (log node).

Fixed a JVM crash caused by frequent garbage collection (GC) when the CDC (log node) runs on extra-large instances.

Fixed interruptions to the CDC (log node) main link that occurred during ALTER operations on table names with backticks ``.

Category

Description

New features

Binlog Dump now filters out delete events from archived tables when generating a Binlog.

Category

Description

Enhancements

Introduced the virtual table feature to map multiple qualifying physical tables to a single virtual table using mapping rules.

Added the daemon_tso_heartbeat_suspend_enabled parameter to control the suspension of TSO service heartbeats under specific conditions.

Optimized data synchronization from binlog to the CCI.

Fixes

Fixed a potential OOM error on the CDC log node, triggered by an oversized binlog event during binlog pipeline recovery.

Category

Description

New features

You can now modify the partition key and sort key of a clustered columnstore index (CCI). This feature improves data management by letting you adapt the CCI's data distribution and sorting strategy to business changes, which optimizes query performance without requiring an index rebuild.

Added support for the in-place rebuilding of a clustered columnstore index (CCI). This operation lets you optimize and reorganize a CCI with no impact on active queries or access to historical data.

Fixed issues

Fixed an issue where a data node (DN) could interrupt the data stream when using the single-stream binary log service.

Category

Description

New feature

The CDC node now filters out delete events from archived tables when generating binary logs.

Category

Description

New feature

CDC (CDC nodes): Multi-stream binary logs now support a one-to-one mapping between a storage node and a binary log stream.

Feature optimizations

Optimized memory management for dump operations on binary logs.

Increased the output rate of binary logs from CDC nodes to downstream systems.

Reduced the load on the global meta service (GMS) by optimizing SQL statements from CDC nodes.

Category

Description

New feature

Change Data Capture (CDC): The multi-stream binary log on log nodes now supports a one-to-one mapping between a storage node and a binary log stream.

Category

Description

New features and optimizations

Added support for CHECK constraints. For details, see CHECK constraints.

Optimized the allocation policy for CDC off-heap direct memory to prevent OOM from insufficient direct memory allocation.

Optimized the forced restart mechanism for CDC timeouts to allow restarts even when the CDC worker process is blocked.

Category

Description

Improvements

Increased the default size of a single binary log file to 1 GB and enabled automatic rotation when the file number exceeds 999,999.

Improved the download logic for transparently consumed binary log files.

Category

Description

Feature optimizations

Optimized the replication logic for primary and secondary nodes.

Optimized the heartbeat timeout detection logic for the CDC high availability (HA) module to prevent unnecessary process restarts.

Fixes

Fixed an issue where an upgrade from an earlier CDC version would be interrupted because of a missing RuntimeVersion field in the system table configuration.

Fixed an issue where the WITH PARSER NGRAM attribute was missing from the output of the SHOW CREATE TABLE statement.

Category

Description

Fix

Fixed a bug causing CDC link interruptions due to compatibility issues when processing REAL and DOUBLE types in CDC metadata.

Category

Description

Fixes

Fixed an issue where a single SQL statement updating multiple tables generated consecutive TableMapEvents, resulting in disordered TraceIDs.

Category

Description

Fixes

Fixed an issue where the CDC process failed to start when a single compute node was unavailable.

Fixed an issue where dump operations failed due to inconsistent offsets between memory and a binary log file.

Category

Description

Fixes

Fixed an issue where the SHOW FULL BINARY LOGS statement returned incomplete information.

Fixed an issue where a binary log file was lost during CDC topology reconstruction due to concurrent uploads of the same file.

Fixed issues where the SQL parser failed to parse ALTER statements with implicit table groups and failed to correctly process marking statements.

Fixed an issue where the replication link could not recover automatically. This failure was caused when the system deleted local files on the master node during CDC Dumper primary-secondary synchronization.

Category

Description

New feature

Enabled CDC (change data capture) nodes to periodically build sync points.

Fixed issue

Fixed an issue where a binary log cursor update failure blocked data consumption.

Category

Description

New features and optimizations

Enabled CDC nodes to periodically build sync points.

Improved full data validation in the upgrade assessment from PolarDB-X 1.0 to PolarDB-X 2.0.

Fixes

Fixed global binary log generation failures caused by metadata changes.

Fixed binary log dump service failures caused by empty requested binary log files.

Category

Description

New features and optimizations

Primary/secondary DDL replication now replicates functions.

Primary/secondary DDL replication now replicates sequences.

Primary/secondary DDL replication now replicates views.

Primary/secondary DDL replication now replicates procedures.

Primary/secondary DDL replication now replicates accounts, roles, and permissions.

Primary/secondary DDL replication now replicates ALTER TABLEGROUP operations.

Primary/secondary DDL replication now replicates ALTER INDEX operations.

Primary/secondary DDL replication now supports consistency-coordinated and aligned replication in multi-stream binary log scenarios.

Primary/secondary replication now enables you to create a replication link based on a timestamp.

Primary/secondary replication now enables you to use SQL commands to verify data consistency between upstream and downstream systems.

A new secondary instance role for primary/secondary replication is now available, enabling instance-level read-only capabilities on the secondary cluster.

Primary/secondary replication now supports full mirror matching of WHERE conditions for DML writes.

You can now bind multi-stream binary log stream names to user names. This eliminates the need to add the 'with' option when executing binary log-related SQL statements with a bound account.

Optimized server_id-based bidirectional replication for primary/secondary replication. You can now specify filter rules from the command line, and the system now automatically checks for server_id anomalies.

Fixes

Fixed an issue where the MULTI ALTER ADD/DROP COLUMN statement caused an incorrect metadata column order, which interrupted the binary log link.

Fixed an issue where the MULTI ALTER ADD/DROP/RENAME INDEX statement could cause index metadata to be lost, which resulted in an incorrect DDL statement being recorded in the binary log.

Fixed a configuration defect in the meta_build_physical_ddl_sql_blacklist_regex parameter that caused ALTER DDL statements to be filtered if they contained the 'user' keyword.

Fixed an issue where DROP INDEX DDL statements were not written to the binary log.

Category

Description

New feature

Added binlog-related O&M commands, such as show full binary logs and show full master status.

Fixed issue

Fixed an issue where exact matching failed in SQL flashback.

Category

Description

New feature

Added the SHOW FULL MASTER STATUS command.

Feature optimization

Improved MySQL binlog dump compatibility to support parameters such as checksum and heartbeat.

Category

Description

New feature

Adds support for SQL flashback to roll back an entire transaction.

Fixed issue

Fixed the recording method for TSO heartbeats in the binary log to resolve crashes in downstream MySQL consumers.

Category

Description

Optimization

Optimized the sleep time for gRPC backpressure control, boosting the external binlog dump speed to 130 MB/s.

Fixes

Fixed an issue that prevented SQL flashback from finishing when no DML or DDL operations occurred within the specified time range.

Category

Description

Optimization

Optimized the SHOW BINARY LOGS statement to resolve slow queries when many binlogs are present.

Fixes

Fixed a buffer overflow when shaping JSON data.

Prevented the metaDataSource scan thread from exiting during an HA switchover in metaDB during a CDC upgrade.

Fixed a deadlock when updating binlog_dumper_info.

Category

Description

New features and optimizations

Added support for the FLUSH LOGS statement.

Optimized the speed of external dumps for transparent consumption.

CDC no longer depends on is_vip for DN connections.

Removed the control dependency for retrieving a MetaDB connection in CDC.

Fixes

Fixed a syntax error in table creation statements with a BIGINT SIGNED column definition.

Fixed an issue where the recover tso generation algorithm could cause gaps in the binary log list.

Fixed an issue where a task dump failed to find the starting offset on the primary instance DN after a read-only instance was scaled.

Category

Description

New feature

Added support for persisting transaction objects to reduce memory usage.

Fixes

Fixed an issue where recover tso could not be smaller than the TSO of the most recent scaling-tagged transaction when recovering based on a historical TSO.

Fixed a bug where the gRPC thread failed to exit when local files on the Dumper Master were deleted during CDC Dumper primary-secondary synchronization.

Category

Description

Fixes

Fixed a link error caused by a heartbeat window conflict when consuming a binary log from OSS.

Fixed a data format conversion error when shaping a TableMapEvent.

Fixed a Sql_Mode value mismatch between the DDL Event and the original value.

Fixed inconsistent metadata between logical and physical tables for nchar and nvarchar types.

Fixed an issue that prevented DDL SQL statements with line feeds from being converted to single-line comments.

Category

Description

New features and optimizations

Added support for monitoring the CPU utilization of the worker process.

Added support for more fine-grained monitoring of Events Per Second (EPS) for INSERT, UPDATE, and DELETE operations.

Optimized the upload speed of binary log files to Object Storage Service (OSS), increasing the rate to 700 MB/s.

Added Operations and Maintenance (O&M) commands for Change Data Capture (CDC), such as start master, stop master, restart master, and reset master.

Fixes

Fixed long blocking during the transparent consumption of offline binary log files.

Fixed an issue where the dump thread would fail to exit if a binary log file was deleted while being read.

Fixed reset master failures caused by excessive physical DDL metadata.

Fixed an inconsistency between the Sql_Mode value in a DDL event and the original sql_mode value.

Fixed a downstream MySQL error caused by an SQL statement containing both ADD PRIMARY KEY and DROP PRIMARY KEY.

Category

Description

Fixes

Fixed an issue where the column order was incorrect after an SQL statement was executed in the SchemaRepository.

Fixed an NPE triggered when data shaping an empty string.

Category

Description

Fix

Fixed an issue where metadata was truncated when it exceeded the length of the mediumtext data type.

Category

Description

Bug fix

Fixed a parsing failure caused by the MySQL charset not being converted to the Java charset.

Category

Description

Feature optimizations

Reduced the RTO of the binary log synchronization link to 15 seconds.

Accelerated CDC metadata recovery for deployments with hundreds of thousands of data tables.

Fixes

Fixed a potential offset calculation error when transparently consuming binary logs.

Fixed a compatibility issue with text(0) during metadata consistency checks.

Category

Description

Fixes

Fixed CDC link interruptions caused by the continuous addition and deletion of primary keys.

Fixed a CDC disconnection that occurred when the consumer program failed to send COM_REGISTER_SLAVE before dumping binary logs.

Fixed a data link interruption in multi-stream data routing when the hash value was Integer.MIN_VALUE.

Category

Description

New features and optimizations

Adds support for multi-level merging. This ensures performance does not significantly decrease as DN nodes scale linearly.

Adds support for transparent consumption. Downstream consumer programs can use the dump protocol to directly consume binary log files archived to OSS without needing to interface with OSS directly.

Adds support for multi-stream binary log. You can customize the number of streams, each with its own independent binary log file.

Adds support for periodically building full metadata snapshots. This prevents historical DDL tagging records from growing excessively.

Adds support for downloading binary log files from OSS in parallel to improve instance recovery speed.

Fixes

Fixed compatibility and correctness issues in shaping binary log event data, improving the stability of the CDC link during DDL changes.

Fixed an issue that could cause dump timeouts for downstream subscriptions after a CDC cluster restart.

Fixed an incorrect latency calculation that occurred during the initial startup of the Dumper process.