Data inconsistency after Redis synchronization
This topic describes how to identify and troubleshoot data inconsistency after Redis synchronization.
Problem description
After you use Data Transmission Service (DTS) to synchronize Redis, data is inconsistent between the source and destination.
Example
After you use Data Transmission Service (DTS) to synchronize Redis data, you run theinfo keyspace command to check the keys on the source and destination. The output shows that the data is inconsistent, with a large proportion of keys having an expiration time.
xxx> info keyspace
# Keyspace
db0:keys=769509,expires=227386,avg_ttl=3426607441
db3:keys=254,expires=0,avg_ttl=0
db2:keys=17,expires=2,avg_ttl=837582
db1:keys=2,expires=0,avg_ttl=0
db15:keys=2,expires=0,avg_ttl=0
[work@rec01 bin]$ ./redis-cli -a 'xxx'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info keyspace
# Keyspace
db0:keys=772864,expires=230736,avg_ttl=3380950183
db1:keys=2,expires=0,avg_ttl=0
db2:keys=17,expires=2,avg_ttl=868931
db3:keys=254,expires=0,avg_ttl=0
db15:keys=2,expires=0,avg_ttl=0
127.0.0.1:6379>Cause
Relying solely on theinfo keyspace command'sexpires value to verify data consistency is misleading because the command provides estimates, not exact values.
-
In the output of the
info keyspacecommand:-
keysindicates the total number of keys. -
expiresindicates the number of keys that have an expiration time, including both unexpired and expired keys.
-
Therefore, when you use theexpires value to check for consistency, keys that have logically expired but have not yet been actively deleted by Redis are included in the count. This leads to the apparent data inconsistency between the source and destination.
Redis removes expired keys under the following conditions:
-
Redis periodically clears expired keys when the CPU is idle.
-
When a key is accessed, Redis checks if it has expired and evicts it if so.
-
During a SCAN operation, Redis checks for and evicts any expired keys it encounters.
For more information about the eviction policy for expired keys in Redis, see How Redis Handles Expired Keys.
Solution
To verify data consistency, use one of the following methods:
When checking for data loss after synchronization, do not rely solely on theexpires statistic, as it includes data that has already expired and makes theexpires value an unreliable measure of consistency.
-
For both the source and destination databases, use the
keyscount and subtract theexpirescount to find the number of permanent keys. If the difference between the results for each database is negligible, the data can be considered consistent. -
Use the redis-full-check tool for verification. For more information, see Use the redis-full-check tool.
How can you make the key counts in theinfo keyspace output appear consistent?
We recommend clearing expired Redis data. The methods for cloud databases and self-managed databases are as follows:
-
Tair (Redis OSS-Compatible): Log on to the Tair (Redis OSS-Compatible) console and use the Clear Data feature. For ApsaraDB for Redis, open the Clear Data dialog box in the console, select the Expired Data tab, choose an execution time (Immediately or Within Maintenance Window), and then click OK. This action runs a SCAN command to bulk-delete all expired data from the instance. This operation is irreversible, and the deleted data cannot be recovered.
-
self-managed Redis database: Use the redis-full-check tool to run a verification. The tool executes commands like SCAN, triggering the cleanup of expired data during the process.