Binlog service

This topic describes the binary log (binlog) service for PolarDB-X, including its two modes and their use cases.

Notes

• The CDC master node replicates binary log files to the secondary node. This ensures that the data is identical on both nodes. Downstream systems can subscribe to and consume data using a filename and position. If a primary/secondary switchover occurs in the CDC component, the filename and offset remain unchanged.

• Distributed transactions are merged only when the transaction policy is set to Timestamp Oracle (TSO). Otherwise, only eventual consistency is guaranteed. The default transaction policy for PolarDB-X is TSO.

• To change the partition key value for a data row, you must perform the operation within a distributed transaction that uses the TSO policy. This ensures that the delete event appears before the insert event in the binary log, which maintains data consistency. To ensure data consistency when you change a partition key value, first use the TSO transaction policy, and then perform one of the following operations:

  • Execute an `UPDATE` SQL statement to modify the partition key value.

  • Execute a `REPLACE` SQL statement to modify the partition key value.

  • Explicitly start a transaction, execute `DELETE`, adjust the partition key value, and then `INSERT` the data again.

Multi-stream mode

Multi-stream binary logging is also fully compatible with the MySQL binlog file format and dump protocol. You can treat each binlog stream as a standalone MySQL instance. For a single log stream, you can execute SQL commands such as CHANGE MASTER and SHOW BINLOG EVENTS to consume or view binlog data.

The multi-stream service is not enabled by default. You must enable it manually in the console. For a single PolarDB-X instance, you can enable multiple multi-stream services at the same time. These services are completely isolated from each other. You can configure each service with a different number of streams, a different data splitting level, and different parameter rules. You can enable different types of multi-stream services as needed.

Splitting levels

Multi-stream binary logging provides three data splitting levels. You can set the level when you enable the multi-stream service to meet the needs of different scenarios.

• Database level

  Data is distributed based on a hash value calculated from the database name. This means binlog data for the same database is always routed to the same binlog data stream. This level is suitable for scenarios where a single PolarDB-X instance has many databases. If transactions do not involve cross-database operations, this policy provides both multi-stream capabilities and transactional integrity.

• Table level

  Data is distributed based on a hash value calculated from the table name. This means binlog data for the same table is always routed to the same binlog data stream. This level is suitable for scenarios with many tables where you want operations on a single table, such as DML and DDL, to remain ordered in the binlog stream.

• Record level

  Data is distributed based on a hash value calculated from the primary key of the data row. This means binlog data for the same data row is always routed to the same binlog data stream. This level is suitable for scenarios where you want to fully discretize data and do not require log data to be ordered by database or table. This policy requires that tables have a primary key. Data from tables without a primary key is discarded.

Data splitting levels support layered configuration at the service level and the database or table level. After a level is configured, it cannot be modified. Modifying the level can cause the same data to drift between different streams, which leads to data consistency issues. Before you enable a multi-stream service, you must evaluate your business scenario and plan the splitting level in advance.

• Service layer

  This is the default splitting level for the multi-stream service. If no splitting level is set for a specific database or table, the default service-level configuration is used.

• Database and table level

  You can set a separate distribution policy for a specific database or table. This policy overrides the service-level configuration to meet specific requirements.

Notes

• After a multi-stream service is created, you cannot modify the number of streams. You must plan the number of streams in advance. The number of streams should be greater than or equal to the number of DNs.

• After a multi-stream service is created, you cannot adjust the splitting level once it is in effect. You must plan this setting in advance.

• If you want to set a separate splitting level for a new data table, you must configure it before any data is written to the table.

• When the splitting level is set to table, the RENAME operation is supported for tables. PolarDB-X always splits data based on the initial table name.

• When the splitting level is set to table, you can set a separate routing rule to prevent data skew. Data skew occurs when data from large tables is concentrated in a few streams.

• To adjust the number of streams or the data splitting level after it is in effect, you can enable a new multi-stream service to replace the current one. This requires O&M adjustments to the downstream consumption pipeline.

• When the splitting level is set to record, data consistency issues may arise if a data table contains a unique key constraint and a unique key swap occurs. For example, a unique key `uk(name)=a` is held first by `id=1` and then by `id=2`. Because the execution order of `delete(id=1,name=a)` and `insert(id=2,name=a)` in the destination database cannot be guaranteed, a write conflict occurs if `insert(id=2,name=a)` is executed before `delete(id=1,name=a)`. In such scenarios, you must set the splitting level to table.

Transparent consumption

CDC first saves binlog files to local disks and supports real-time uploads to remote storage, such as OSS. Files on local disks typically have a short lifecycle. Files on remote storage have a longer lifecycle, for example, 15 days. For files on remote storage, CDC provides a transparent consumption feature. This feature abstracts away the storage differences between local and remote locations. Downstream systems do not require any additional adaptation to access binlog data on remote storage.

Active geo-redundancy

In addition to importing data into external systems, you can use the CDC of PolarDB-X to implement active geo-redundancy. For example, you can partition user data by region across different data centers. In this setup, write operations are directed to a specific data center, while read operations can access the nearest replica. This replica data is synchronized from the source data center using CDC.