Tiered storage (data archiving)

Feature overview

Tiered storage deeply integrates the WiredTiger storage engine with OSS. At the collection level, tiered storage uploads all data files of a collection from local disks to cold storage in OSS. After a collection is archived, the system releases the local storage space occupied by the collection and retains the collection metadata and index definitions so that the data can be restored when the collection is unarchived.

Tiered storage is suitable for the following scenarios:

• Log data must be retained at low cost after the active query period ends.

• Historical orders, transaction records, and other data must be retained for an extended period to meet compliance requirements but are infrequently accessed.

• Business data continues to grow, resulting in insufficient local disk capacity and a need to free up storage space.

Prerequisites

Before you use tiered storage, make sure that your instance meets the following requirements.

Apply for tiered storage

Tiered storage requires a specific minor version. Self-service upgrade to this version is not yet available from Minor Version Upgrade in the console. Technical support must confirm instance eligibility and upgrade the target instance.

1. Make sure that the target instance meets the prerequisites.

2. Prepare the ID of the instance for which you want to enable the feature, such as dds-bp1xxxxxxxxxxxxx. If you want to enable the feature for multiple instances, prepare a list of instance IDs.

3. Submit a ticket or contact your technical support representative to apply for the feature, and provide the following information:

   • Instance ID. This field is required. You can provide multiple instance IDs.

   • The region of the instance.

   • The preferred time window for enabling the feature.

4. Wait for technical support to grant the instance eligibility for the feature and upgrade the instance to a minor version that supports tiered storage.

   Important

   The upgrade triggers a rolling restart and a switchover between the primary and secondary nodes. Connections may be interrupted for a few seconds. Make sure that your application can automatically reconnect, and ask technical support to schedule the upgrade during off-peak hours.

5. After the feature is enabled, verify that Data Archiving is available for the instance.

Enable tiered storage

1. Log on to the ApsaraDB for MongoDB console.

2. In the left-side navigation pane, choose Replica Set Instances and find the target instance.

3. Click the ID of the target instance to go to the Basic Information page.

4. In the left-side navigation pane, choose Data Archiving.

5. When you use the feature for the first time, click Enable Tiered Storage.

   The system automatically creates an OSS bucket and configures authorization. The feature typically takes 1 to 2 minutes to enable. During this period, the instance continues to provide services as expected and your existing workloads are not affected.

6. Verify that the Configure Tiered Storage and Archived Cold Data tabs appear on the page.

   The two tabs display collections in local storage and collections archived to OSS, respectively.

Archive a collection

Archive a local collection to cold storage in OSS to free up local disk space.

1. On the Data Archiving page, click the Configure Tiered Storage tab.

2. Select the target collection from the collection list.

   Note

   You can archive only one collection at a time.

3. Click Archive, confirm the collection information, and then click OK.

4. View the archiving progress in Task Center.

   During archiving, the target collection can be read but not written.

5. After archiving is complete, verify that the collection is moved from the hot data list to the cold data list.

   The system releases the local storage space occupied by the collection.

Unarchive a collection

An archived collection can be read but not written. To insert, update, or delete data in the collection, restore the cold data to local storage so that the collection supports normal read and write operations.

1. On the Data Archiving page, click the Archived Cold Data tab.

2. Select the target collection from the list of archived collections.

3. Click Unarchive, confirm the collection information, and then click OK.

4. View the unarchiving progress in Task Center.

   During unarchiving, the target collection is inaccessible.

5. After unarchiving is complete, verify that the collection is moved from the cold data list to the hot data list and supports normal read and write operations.

Collection read and write capabilities

The read and write capabilities of a collection vary based on its state. Before you perform an operation, make sure that your workloads can tolerate the corresponding limits.

View the archiving status

On the Data Archiving page, use the Configure Tiered Storage and Archived Cold Data tabs to view the distribution status of collections. The page displays information such as database names, collection names, and data sizes. Collections that are being archived or unarchived display task progress.

Limits

Precautions

• Version upgrade: Upgrading to a minor version that supports tiered storage triggers a rolling restart and a switchover between the primary and secondary nodes. Make sure that your application can automatically reconnect and reserve an upgrade window.

• Write restrictions after archiving: After an archiving task starts, the target collection immediately becomes read-only. All write operations (insert, update, and delete) and DDL operations such as creating indexes on the collection fail. Before you archive a collection, make sure that the collection no longer receives write traffic. If your workloads are still writing to the collection, redirect or stop the write traffic first.

• Access restrictions during unarchiving: During unarchiving, the collection is completely inaccessible, and all read and write operations fail. After unarchiving is complete, the collection automatically supports normal read and write operations. No additional operations are required.

• Unarchiving duration: The bandwidth for downloading data from OSS is lower than the bandwidth for uploading data to OSS. In addition, the restoration process involves rebuilding and verifying WiredTiger data files. Therefore, unarchiving typically takes three to five times as long as archiving. Before you archive a collection, evaluate whether your workloads can tolerate several hours of downtime when the collection is unarchived in the future.

• Operation window: Archive and unarchive collections during off-peak hours to minimize the impact of tasks on the read and write performance of other collections in the instance.

• Task exceptions: If an archiving or unarchiving task does not make progress for an extended period or fails, submit a ticket or contact technical support.