Use MongoShake to migrate data from a Serverless instance to a replica set or sharded cluster instance-ApsaraDB for MongoDB(MongoDB)-阿里云帮助中心

MongoShake is an open source tool developed by Alibaba Cloud. It is used to migrate and synchronize data between MongoDB databases. This topic describes how to use MongoShake to migrate data from a MongoDB Serverless instance to a MongoDB replica set or sharded cluster instance.

Impacts

The migration task consumes some read and write throughput CUs of the source instance. For more information about read and write throughput CUs, see Read and write throughput CUs.
During the migration, read or write operations on the source instance may become unresponsive.

Database account permissions

Instance	Permissions
Source MongoDB Serverless instance	readWrite permission.
Destination MongoDB replica set or sharded cluster instance	readWrite permission.

For more information about how to create a database account and grant permissions, see Use DMS to manage MongoDB database users.

Preparations

For optimal replication performance, make sure that the network type of the source MongoDB Serverless instance is virtual private cloud (VPC). If the instance uses the classic network, switch it to a VPC. For more information, see Switch from the classic network to a VPC.
Create a MongoDB replica set or sharded cluster instance to serve as the synchronization destination. When you create the instance, select the same VPC as the source MongoDB Serverless instance to minimize network latency. For more information, see Create a replica set instance or Create a sharded cluster instance.
Note If the destination is a sharded cluster instance, you must enable sharding for the destination database and set a shard key for the collection. For more information, see Configure data sharding to maximize shard performance.
Create an ECS instance to run MongoShake. When you create the instance, select the same VPC as the source MongoDB Serverless instance to minimize network latency. For more information, see Create an ECS instance.
Add the private IP address of the ECS instance to the whitelists of the source and destination MongoDB instances. Make sure that the ECS instance can connect to both MongoDB instances. For more information, see Modify a whitelist.

Note If your network does not meet these requirements, you can request public endpoints for the source and destination MongoDB instances. Then, add the public IP address of the ECS instance to the whitelists of both instances and perform the synchronization over the public network. For more information, see Request a public endpoint and Modify a whitelist.

Procedure

The following procedure describes the migration steps. An ECS server that runs the Alibaba Cloud Linux operating system and is accessed over a private network is used as an example.

Log on to the ECS instance. For more information, see Use Workbench to log on to a Linux instance.
Run the following command to download the latest version of the MongoShake program.
```
wget <URL_of_the_latest_MongoShake_package>
```
Example:
```
wget https://github.com/alibaba/MongoShake/releases/download/release-v2.6.4-20210414/mongo-shake-v2.6.4_2.tar.gz
```
Note The download URL in the example is for MongoShake 2.6.4_2. For the latest version, see MongoShake releases.

Run the following command to decompress the MongoShake package.

tar xvf <MongoShake_package_file_name>

Example:

tar xvf mongo-shake-v2.6.4_2.tar.gz

Run the following command to change to the directory where the collector.conf file is located.
```
cd the directory where the collector.conf configuration file is located
```
Example:
```
cd mongo-shake-v2.6.4
```

Run the following command to modify the MongoShake configuration file collector.conf.

vi collector.conf

In the vi editor, press i. Modify the parameter values based on the descriptions in the configuration file and your business needs.

Configure the following parameters as described.

Parameter	Description	Example
sync_mode	The data synchronization mode. The default value is `incr`. Set this parameter to `all`. `all`: Performs full data synchronization and incremental data synchronization. `full`: Performs only full data synchronization. `incr`: Performs only incremental data synchronization.	`sync_mode = all`
mongo_urls	The connection string URI of the source MongoDB Serverless instance. Note Use a private endpoint for the connection to minimize network latency. For more information about the connection string URI format, see Connection description for Serverless instances.	`mongo_urls = mongodb://user46436397:******@dds-bpxxxxxxxx.mongodb.rds.aliyuncs.com:3717/admin` Note** The password cannot contain the at sign (@). Otherwise, the connection fails.
tunnel.address	The connection string URI of the destination MongoDB instance. Note Use a private endpoint for the connection to minimize network latency. For more information about the connection string URI format, see Connection description for replica set instances or Connection description for sharded cluster instances.	`tunnel.address = mongodb://root:**@dds-bpxxxxxxx.mongodb.rds.aliyuncs.com:3717,dds-bpxxxxxxxx.mongodb.rds.aliyuncs.com:3717/admin?replicaSet=mgset-xxxxxx` Note** The password cannot contain the at sign (@). Otherwise, the connection fails.
filter.ddl_enable	Specifies whether to enable DDL synchronization. The default value is `false`. Set this parameter to `true`. `true`: enabled `false`: disabled	`filter.ddl_enable = true`
full_sync.reader.collection_parallel	The maximum number of collections that MongoShake can pull concurrently at a time. The default value is 6. Set this parameter to 1.	`full_sync.reader.collection_parallel = 1`
full_sync.reader.write_document_parallel	The number of concurrent threads for writing data to a single collection. The default value is 8. Set this parameter to 4.	`full_sync.reader.write_document_parallel = 4`
full_sync.create_index	Specifies whether to create indexes after synchronization is complete. Set this parameter to `foreground`. `foreground`: Creates foreground indexes. `background`: Creates background indexes. `none`: Does not create indexes.	`full_sync.create_index = foreground`
incr_sync.mongo_fetch_method	The method to pull data from the database. The default value is `oplog`. Set this parameter to `change_stream`. `oplog`: Pulls oplogs from the source. `change_stream`: Pulls change events from the source. This method is supported only for MongoDB 4.0 and later. For DDL synchronization, only the creation, deletion, and renaming of databases and tables are supported.	`incr_sync.mongo_fetch_method = change_stream`
special.source.db.flag	Support for special forms. Set this parameter to `aliyun_serverless`.	`special.source.db.flag = aliyun_serverless`

After you modify the file, press the Esc key, enter :wq!, and press Enter.

Run the following command to start the migration task.
```
./collector.linux -conf=collector.conf &
```
Note The migration speed may be slow because of the read and write throughput CU limits on the MongoDB Serverless instance. Wait for the migration to complete.
Run the following command to change to the `logs` directory.
```
cd logs
```
Run the following command to view the log information.
```
vi collector.log
```
When the following log entry appears, the full data migration is complete.
Note If you enabled incremental migration, this means that the full data migration is complete and the task has entered the incremental migration phase.
```
[2021/04/30 14:20:38 CST] [INFO] ------------------------full sync done!------------------------
```
After you view the log, press the Esc key, enter :q, and press Enter.

Check whether the incremental data migration is complete.

Run the following command to change to the installation directory of the MongoShake program.
```
cd <MongoShake_installation_directory>
```
Example:
```
cd mongo-shake-v2.6.4
```

Run the following command to view the time values of lsn_ckpt and now.

curl -s http://127.0.0.1:9100/repl | python -m json.tool

A response similar to the following is returned. For descriptions of the parameters, see Field descriptions.

{
    "log_size_avg": "320.00B",
    "log_size_max": "332.00B",
    "logs_get": 5,
    "logs_repl": 5,
    "logs_success": 5,
    "lsn": {
        "time": "2021-05-21 14:18:37",
        "ts": "6964624121430802437",
        "unix": 1621577917
    },
    "lsn_ack": {
        "time": "2021-05-21 14:18:37",
        "ts": "6964624121430802437",
        "unix": 1621577917
    },
    "lsn_ckpt": {
        "time": "2021-05-21 14:19:19",
        "ts": "0",
        "unix": 0
    },
    "now": {
        "time": "2021-05-21 14:19:19",
        "unix": 1621577959
    },
    "replset": "default-0",
    "tag": "$",
    "tps": 0,
    "who": "mongoshake"
}

When the time values of lsn_ckpt and now are almost the same, the incremental data migration is complete.

Field descriptions

Field	Description
`logs_get`	The total number of oplogs pulled.
`logs_repl`	The total number of oplogs that the system tried to write to the destination database.
`logs_success`	The total number of oplogs successfully written to the destination database.
`lsn`	The checkpoint time for pulling data. This is the initial time when no data is being written.
`lsn_ack`	The checkpoint time for data successfully written to the destination database. This time changes as data is written.
`lsn_ckpt`	The checkpoint time for data successfully written to the destination database. This time is stable and persistent.
`now`	The current time.
`replset`	The name of the source.