Data migration

Step 1: Configure and start data synchronization

1. Unzip NimoShake.zip, navigate to the NimoShake directory, and edit the shared configuration file conf/nimo.conf. The nimo-shake, nimo-full-check, and nimo-repair tools share this file, and each tool reads only the parameters it requires. The following are the core configuration parameters required to start the synchronization:

   Parameter	Description	Example value
   sync_mode	Synchronization mode: all indicates full and incremental, and full indicates full only.	all
   source.access_key_id	The AccessKey ID of the source AWS DynamoDB.	AKIAIOSFODNN7...
   source.secret_access_key	The secret access key of the source AWS DynamoDB.	wJalrXUtnFEMI...
   source.region	The region where the source AWS DynamoDB resides.	cn-north-1
   target.endpoint_url	The DynamoDB access endpoint of the target PolarDB cluster (including the port number).	http://pe-xxx.rwlb.rds...
   target.access_key_id	The AccessKey ID of the DynamoDB account for the target PolarDB cluster.	your-polardb-access-key
   target.secret_access_key	The secret access key of the DynamoDB account for the target PolarDB cluster.	your-polardb-secret-key
   filter.collection.white	A whitelist for table filtering. Separate multiple table names with a semicolon (;). This parameter cannot be used with the blacklist parameter.	c1;c2
   filter.collection.black	A blacklist for table filtering. Separate multiple table names with a semicolon (;). This parameter cannot be used with the whitelist parameter.	c1;c2

   To use the recommended S3 snapshot-based check mode (see Step 2), you must also configure these native PostgreSQL connection parameters:

   Parameter	Description	Example value
   s3.export_state_file	The path to the export state file. This file is written by nimo-shake and read by nimo-full-check.	../nimo-shake-s3-exports.json
   s3.export_bucket	The name of the target S3 bucket for DynamoDB data export.	my-export-bucket
   s3.export_prefix	The S3 path prefix used to organize exported files.	exports/my-project/
   target.pg.endpoint_url	The native port address of PolarDB for PostgreSQL (host:port format).	pc-xxx.pg.polardb.rds...:5432
   target.pg.user	The username for PolarDB for PostgreSQL (also used as the schema name). Note A DynamoDB account is essentially a standard database account. Its secret access key (SK) is generated by encrypting the password you set with a compatible algorithm. Therefore, target.pg.user and target.access_key_id correspond to the same account. target.pg.password is your custom password, and the SK is a key derived from this password.	your-username
   target.pg.password	The password for PolarDB for PostgreSQL.	your-password
   target.pg.check_database	The name of the dedicated database that stores the incremental check tables. During the incremental synchronization phase, nimo-shake writes the primary keys of changed records to the incremental check tables in this database for the s3 or incr check modes to read. You must manually create this database before startup, and nimo-shake and nimo-full-check must be configured with the same value. Note target.pg.check_database points to a dedicated database that stores incremental check tables. Before you start the synchronization, use a Privileged Account to manually create the database and set its owner to **target.pg.user** (the DynamoDB account described above). This way, the account has the permissions to create schemas and incremental check tables in the database.	nimoshake_check

   (Optional) Full synchronization in PostgreSQL direct-write mode

   By default, full synchronization uses the native PostgreSQL direct write mode to directly write data to PolarDB by using the standard PostgreSQL protocol, bypassing the DynamoDB compatibility layer API for higher performance. To revert to the DynamoDB layer API (BatchWriteItem, which writes a maximum of 25 items at a time), you can manually change this parameter to dynamodb. You can configure the following parameters in the [SHAKE] section of the conf/nimo.conf file:

   Parameter	Description	Default
   full.write_protocol	The protocol used for full data writes. postgresql (default): the high-performance PostgreSQL direct-write mode. dynamodb: the DynamoDB layer API (BatchWriteItem, with a maximum of 25 items per batch).	postgresql
   full.document.write.batch	The number of documents to write in each batch. When you use the PostgreSQL direct-write mode, the batch size is not limited to 25 documents. You can increase this value to further improve write throughput. If you switch back to the dynamodb protocol, change this value back to 25.	1000
   full.document.concurrency	The number of concurrent write threads per table. In PostgreSQL direct-write mode, the tool automatically sets the connection pool size based on this value to ensure a one-to-one correspondence between threads and connections.	4

   Note

   • PostgreSQL direct-write mode also requires you to configure the four target.pg.* parameters. This set of configurations is shared with the s3 / incr check modes, so you do not need to enter them again.

   • This mode affects only the write path during the full synchronization phase. Incremental synchronization always writes data through the DynamoDB-compatible API layer.

2. (Optional) Stream Buffer protection for long-running full synchronization

   AWS DynamoDB Streams retains change records for only 24 hours. When the source table is large and full synchronization is estimated to take longer than 24 hours, incremental changes generated during the full phase may be automatically discarded by AWS if they remain in Streams for more than 24 hours, causing data loss in incremental synchronization. To address this, you can enable the Stream Buffer: the incremental listener writes Streams change records to a local buffer in real time from the start of full synchronization, bypassing the 24-hour retention limit. After full synchronization completes, the buffered changes are consumed and replayed from the local buffer to ensure no changes are lost. This feature is disabled by default and should be enabled only when the estimated full synchronization duration approaches or exceeds 24 hours. Configure the following parameters in conf/nimo.conf:

   Parameter	Description	Default
   incr.stream_buffer.enable	Master switch for the Stream Buffer.	false
   incr.stream_buffer.dir	Buffer file directory. If left empty, defaults to a sibling directory of the checkpoint (<workdir>/<id>-buffer).	Empty
   incr.stream_buffer.max_seg_size_mb	Maximum size (MB) of a single buffer segment file. Exceeding this value triggers rotation.	64
   incr.stream_buffer.max_seg_count	Maximum number of records per buffer segment. Exceeding this value triggers rotation.	10000
   incr.stream_buffer.max_seg_duration_sec	Time interval (seconds) for forced segment rotation.	300
   incr.stream_buffer.max_size_mb	Global disk quota (MB), shared by all shards and tables (not per-shard). Plan based on available disk space on the machine.	51200 (50 GB)
   incr.stream_buffer.quota_backoff_sec	When the buffer exceeds the global quota, the writer logs an ERROR every 30 seconds to prompt operations to expand disk space or accelerate full synchronization. After the duration specified by quota_backoff_sec elapses, the synchronization program automatically terminates.	1800

   • Disk capacity can be roughly estimated by the formula: Required disk (MB) ≈ Full sync duration (seconds) × Peak writes (records/second) × (Average bytes per record / 3) / 1 MB × 1.5. Always over-provision disk space rather than under-provision.

   • During full synchronization, the reader does not consume the buffer. Therefore, once the buffer exceeds max_size_mb, it cannot drain itself. In this case, the process crashes after backoff. You must increase disk space or improve full synchronization speed before restarting.

3. Run the binary file for your operating system to start the synchronization.

   Note

   We recommend running the task with nohup in the background to prevent the synchronization task from being interrupted if the terminal session disconnects.

   # Execute in the NimoShake directory
   nohup ./bin/nimo-shake.linux.amd64 -conf=./conf/nimo.conf > /dev/null 2>&1 &

   The program first performs a full synchronization. After the full synchronization is complete, the program automatically switches to the incremental synchronization phase and continues to run.

Step 2: Check data consistency

1. nimo-full-check and nimo-shake share the same conf/nimo.conf file, so you do not need to switch directories or edit extra configuration files. Select a check mode using the mode parameter:

   • scan: Performs a real-time scan to compare data between the source and the target. This mode is the simplest to configure and requires no additional infrastructure, but it continuously consumes DynamoDB read capacity during the check.

   • s3 (Recommended): When full synchronization is complete, nimo-shake automatically exports the source data to S3 and creates a point-in-time snapshot. nimo-full-check uses this snapshot as the baseline for the check, which avoids rescanning the source DynamoDB. This requires the S3-related parameters from Step 1.

   • incr (Use with s3 mode): During the incremental synchronization phase, nimo-shake writes the primary key of each changed record into the incremental check table (ct_{user}_{table}) in PolarDB. nimo-full-check reads the incremental check table and checks only the changed records.

   • Comparison of the two check methods

     	Scan mode	S3 / incr mode
     Required downtime	Longer. Business writes must be stopped during the check to prevent the source data from changing during the scan.	Shorter. The check is based on a snapshot. After stopping writes, you only need to confirm that incremental data has caught up before the cutover.
     Impact on the source	Continuously consumes DynamoDB read capacity.	The full check is based on an S3 snapshot and does not consume DynamoDB capacity. The incremental check (incr mode) queries only changed records, which consumes less capacity.
     Configuration complexity	Low.	Higher. It requires additional configuration for S3 and PolarDB for PostgreSQL connections.

2. (Optional) Incremental continuous verification (drain mode, disabled by default): By default, the incr mode performs a one-time verification of all accumulated change records in the incremental check table during the cutover downtime window. When incremental synchronization runs for an extended period (hours to days), the incremental check table accumulates a large number of pending primary keys, making the final verification at cutover time-consuming. After you enable incremental continuous verification (drain), nimo-full-check mode=incr runs as a long-lived process: during incremental synchronization, it continuously verifies and consumes records that have been confirmed consistent from the incremental check table, keeping the incremental check table small before cutover. At cutover time, only a minimal number of remaining records need final verification, significantly reducing the verification time within the downtime window. This feature is disabled by default and does not affect any existing behavior. Enable it only when the incremental period is long and cutover downtime is critical.

   Important

   • The check_drain.enable switch applies to both nimo-shake and nimo-full-check. Both tools share the same conf/nimo.conf file, so a single configuration takes effect for both.

   • Before enabling this feature in production, we recommend completing a full traffic verification on a test instance.

   Configure the following parameters in conf/nimo.conf (no changes needed by default):

   Parameter	Description	Default
   check_drain.enable	Master switch for incremental continuous verification (drain). Applies to both nimo-shake and nimo-full-check. When set to false (default), the behavior is identical to the default behavior (incr mode runs once and exits).	false
   check_drain.finalize_flag_file	Path to the finalize sentinel file. At cutover time, operations run touch on this file, and the long-lived verification process enters the final verification phase in the next cycle, produces the authoritative difference report, and exits. Required when enable=true.	Empty
   check_drain.check_qps	Maximum BatchGetItem requests per second for drain verification (independent limits for source and target). This parameter limits the pressure drain places on the source and target to avoid competing with incremental synchronization resources. Increase this value if verification is too slow.	20

   Note

   • In addition to the parameters listed above, conf/nimo.conf also contains advanced tuning parameters such as check_drain.cycle_interval_sec, check_drain.settle_sleep_ms, and check_drain.consistent_read. Keep the default values in most cases. See the inline comments in the configuration file for details.

   • After you enable drain, the cutover phase no longer requires running nimo-full-check again. Instead, run touch on the finalize sentinel file to trigger the long-lived process to complete the final verification and produce the final difference report. For more information, see Perform business cutover - Last consistency check.

   • The long-lived process for incr mode should be started after the s3 baseline verification completes. The remaining verification process is the same as the default.

3. Start the check task.

   nohup ./bin/nimo-full-check.linux.amd64 -conf=./conf/nimo.conf > /dev/null 2>&1 &

   The check tool stores detailed logs and data difference reports in the default directories logs/ and nimo-full-check-diff/, respectively. The following is sample output from a successful check:

   [INFO] documentChecker[0] with table[{nimo-shake src2}] [success: 91, fail: 0] item matches!
   [INFO] documentChecker[2] with table[{nimo-shake src1}] [success: 96, fail: 0] item matches!
   [INFO] documentChecker[0] with table[{nimo-shake src2}] [success: 92, fail: 0] item matches!
   [INFO] documentChecker[2] with table[{nimo-shake src1}] [success: 97, fail: 0] item matches!
   [INFO] documentChecker[0] with table[{nimo-shake src2}] [success: 93, fail: 0] item matches!
   [INFO] documentChecker[0] with table[{nimo-shake src2}] [success: 94, fail: 0] item matches!
   [INFO] documentChecker[2] with table[{nimo-shake src1}] [success: 98, fail: 0] item matches!
   [INFO] documentChecker[0] with table[{nimo-shake src2}] [success: 95, fail: 0] item matches!
   [INFO] documentChecker[2] with table[{nimo-shake src1}] [success: 99, fail: 0] item matches!
   [INFO] documentChecker[0] with table[{nimo-shake src2}] [success: 96, fail: 0] item matches!
   [INFO] documentChecker[2] with table[{nimo-shake src1}] [success: 100, fail: 0] item matches!
   [INFO] documentChecker[2] with table[{nimo-shake src1}] check completed. real total count: 100
   [INFO] documentChecker[2] with table[{nimo-shake src1}] close executor
   [INFO] documentChecker[2] finishes checking table[src1]
   [INFO] documentChecker[0] with table[{nimo-shake src2}] [success: 97, fail: 0] item matches!
   [INFO] documentChecker[0] with table[{nimo-shake src2}] [success: 98, fail: 0] item matches!
   [INFO] documentChecker[0] with table[{nimo-shake src2}] [success: 99, fail: 0] item matches!
   [INFO] documentChecker[0] with table[{nimo-shake src2}] [success: 100, fail: 0] item matches!
   [INFO] documentChecker[0] with table[{nimo-shake src2}] check completed. real total count: 100
   [INFO] documentChecker[0] with table[{nimo-shake src2}] close executor
   [INFO] documentChecker[0] finishes checking table[src2]
   [INFO] all documentCheckers finish
   [INFO] checker finishes!
   [INFO] full-check completes!

4. (Optional) Repair inconsistent data:

   1. When data in the source DynamoDB is inconsistent with the data in the target PolarDB for PostgreSQL cluster, the inconsistent tables are listed in the specified diff directory. Example:

      Note

      The default diff directory is nimo-full-check-diff. You can configure it in conf/nimo.conf by using the diff_output_file parameter.

      nimo-full-check-diff/
        └── testtable-0
        └── testtable-1

   2. Run the repair task:

      nohup ./bin/nimo-repair.linux.amd64 -conf=./conf/nimo.conf > /dev/null 2>&1 &

Step 3: (Optional) Configure reverse synchronization

Before the business cutover, you can configure a reverse synchronization link from PolarDB for PostgreSQL to the source DynamoDB. This link is activated during the official switch to PolarDB to enable data backflow and provide a safeguard for business rollbacks. The configuration procedure is as follows:

Step 4: Perform business cutover

Once incremental synchronization latency is low and the data consistency check passes, you can plan the business cutover. When you are ready, follow these steps:

1. Final check: Before the planned downtime window, run the consistency check tool repeatedly until you confirm that incremental synchronization latency is minimal and the number of data discrepancies is zero or within an acceptable range.

2. Stop source writes: At the start of the downtime window, stop all applications from writing to the source AWS DynamoDB.

3. Wait for synchronization to complete: Monitor the nimo-shake logs to confirm that no new incremental data needs to be synchronized.

4. Last consistency check: Run the nimo-full-check tool again to ensure that the data in the source and target is completely consistent.

   Note

   If you have enabled incremental continuous verification (drain mode), you do not need to start nimo-full-check again in this step. The long-lived verification process has been continuously verifying and consuming the incremental check table during the incremental phase. After you confirm that the nimo-shake incremental write rate has dropped to zero (the queue is drained), create the finalize sentinel file (the file specified by check_drain.finalize_flag_file, for example, touch /tmp/nimo-finalize). The long-lived process automatically enters the final verification phase, produces the authoritative difference report, and exits.

5. Stop the synchronization tool: After you confirm that the data is fully synchronized, stop the nimo-shake process.

6. Switch application connections: Stop the business applications and change their database connection configurations from the AWS DynamoDB endpoint to the DynamoDB access endpoint of PolarDB.

7. (Optional) Start the reverse synchronization task: During the official business switch to PolarDB, start the reverse synchronization task. This task uses Realtime Compute for Apache Flink to implement reverse data replication, which provides a safeguard for business rollbacks.

8. Start business applications: Restart the business applications. The cutover is complete.

9. (Optional) Stop the reverse synchronization task: After the cutover is complete and the business is running stably for a period, you can stop the reverse synchronization task (which includes stopping the Flink job and releasing its resources).