Verify migrated data

Prerequisites

• The data migration is complete.

• An ECS instance that runs a Linux operating system is created. For more information, see Create an ECS instance.

• The ECS instance can access both the source and destination Redis instances.

  Note

  • If the ECS instance and the Redis instance are in the same VPC and availability zone, add the internal IP address of the ECS instance to the Redis whitelist. For more information, see Configure an IP whitelist.

  • If the ECS instance and the Redis instance are not in the same VPC and availability zone, you can connect to the Redis instance by using its public endpoint. For more information, see Connect to a Tair (Redis OSS-compatible) instance using a public endpoint.

Use cases

Errors during data migration can cause inconsistencies between the source and destination instances. You can use redis-full-check to find these discrepancies and correct the data.

redis-full-check is an Alibaba Cloud-developed Redis data verification tool. It extracts data from the source and destination, performs multiple rounds of comparisons, and records any differences in an SQLite3 database.

Procedure

1. Connect to the ECS instance that can access your Tair (Redis OSS-compatible) instance. For more information, see Connect to an ECS instance.

2. Download the redis-full-check tool.

   Note

   We recommend downloading the latest release.

3. Decompress the redis-full-check.tar.gz file.

   tar -xvf redis-full-check.tar.gz

4. Run the following command to perform data verification:

   ./redis-full-check -s "<source_endpoint_1:port;source_endpoint_2:port;...>" -p <source_password> -t <destination_endpoint:port> -a <destination_password> --m=1 --sourcedbfilterlist=0;2 --targetdbfilterlist=0;2

   The following is a sample command:

   ./redis-full-check -s "10.xx.xx.1:6379" -p SourcePwd233 -t "r-bp1zc********.redis.rds.aliyuncs.com:6379" -a testaccount:Test1234 --m=1 --sourcedbfilterlist=0;2 --targetdbfilterlist=0;2

   For a complete list of parameters, see redis-full-check Configuration Guide.

   Table 1. Common redis-full-check parameters

   Parameter	Description	Example
   -s	The endpoint and port of the source Redis instance. Note If the source is a cluster instance, separate multiple endpoints with a semicolon (;). Enclose the endpoints in double quotation marks (""). This parameter is required.	r-bp1xxxxxxxxxxxxx.redis.rds.aliyuncs.com:6379 "10.xx.xx.1:7000;10.xx.xx.1:7001;10.xx.xx.2:7002;10.xx.xx.2:7003"
   -p	The credentials for the source Redis instance, in <username>:<password> format. Example: testaccount:Test1234. If the instance does not require a username, specify only the password. If the instance does not require a password, this parameter can be omitted.	SourcePwd233 or testaccount:Test1234
   -t	The endpoint and port of the destination Redis instance. Note If the destination is a cluster instance, separate multiple endpoints with a semicolon (;). Enclose the endpoints in double quotation marks (""). This parameter is required.	r-bp1xxxxxxxxxxxxx.redis.rds.aliyuncs.com:6379 "10.xx.xx.1:7000;10.xx.xx.1:7001;10.xx.xx.2:7002;10.xx.xx.2:7003"
   -a	The credentials for the destination Redis instance, in <username>:<password> format. Example: testaccount:Test1234. If the instance does not require a username, specify only the password. If the instance does not require a password, this parameter can be omitted.	TargetPwd233 or testaccount:Test1234
   --sourcedbtype	The type of the source instance. Valid values: 0 (Default): standalone or standard 1: cluster (direct connection mode) 2: cluster (proxy mode)	--sourcedbtype=1
   --sourcedbfilterlist	The databases to verify on the source Redis instance. Note This parameter is not required for Redis OSS cluster instances. For other cluster instance types, all databases are verified by default. To specify multiple databases, separate their numbers with a semicolon (;).	--sourcedbfilterlist=0;1;2
   --targetdbtype	The type of the destination instance. Valid values: 0 (Default): standalone or standard 1: cluster (direct connection mode) 2: cluster (proxy mode)	--targetdbtype=0
   --targetdbfilterlist	The databases to verify on the destination Redis instance. Note This parameter is not required for Redis OSS cluster instances. For other cluster instance types, all databases are verified by default. To specify multiple databases, separate their numbers with a semicolon (;).	--targetdbfilterlist=0;1;2
   -d	The output filename for the list of inconsistent data. Default: result.db.	xxx.db
   --comparetimes	The number of verification rounds. Default: 3. Minimum: 1. We recommend a value of 5 or less.	--comparetimes=1
   -m	The verification mode. Valid values: 1: full verification. 2 (Default): Verifies value lengths only. 3: Verifies key existence only. 4: Performs a full verification but ignores large keys.	-m=1
   --qps	The queries-per-second (QPS) limit. The default value is 15000. Note The minimum value is 1. The maximum value depends on your server's performance.	--qps=10
   --filterlist	A list of keys to compare, separated by vertical bars (|). Note abc*: Matches all keys that start with "abc". abc: Matches only the key "abc".	--filterlist=abc*|efg|m*

   Note

   When the command finishes, the tool prints a summary to the command line. The example output indicates two inconsistent keys. A value of 0 means the data is consistent.

   all finish successfully, totally 2 keys or fields conflict

5. View the SQLite3 database containing the inconsistent keys.

   1. Run sqlite3 result.db.3.

      Note

      By default, the list of inconsistent data is saved to result.db.3.

   2. Run SELECT * FROM key;.

      Figure 1. View the list of inconsistent data

      [root@redis-server RedisFullCheck]# sqlite3 result.db.3
      SQLite version 3.7.17 2013-05-20 00:56:22
      Enter ".help" for instructions
      Enter SQL statements terminated with a ";"
      sqlite> SELECT * FROM key;
      1|differentkey1|string|lack_target|0|10|0
      2|differentkey2|list|lack_target|0|4|0
      sqlite>

      Note

      The SQLite3 database contains two tables: key and field.

      • The key table stores the inconsistent keys.

      • For hash, set, zset, and list data types, the field table stores details about the inconsistent data.