Archive data to dedicated storage

Billing

This feature is free of charge during the public preview. You will be notified in advance if this feature becomes a paid service.

Usage notes

• There is no limit on the amount of data that you can archive to dedicated storage.

• The archiving task may fail if a source table has a composite primary key of three columns or has three or more unique keys. We recommend that you fully test these source tables before you archive data. You can also use other archiving methods, such as DBS backups.

• If the source database is MongoDB, the structure of the archived data may have issues. You must check and fix any issues in the archived data before you clean up the temporary tables.

• When you archive data from MongoDB to dedicated storage, unstructured data is converted to structured data. Before you start the archiving task, make sure that the field names, number of fields, and data types are consistent for all documents in the same MongoDB collection. This ensures that you can query the data in dedicated storage after archiving. Otherwise, queries may fail.

Archive data

1. Log on to the DMS console V5.0.

2. In the top navigation bar, choose Solution > Data Archiving.

   Note

   If you use the DMS console in simple mode, move the pointer over the icon in the upper-left corner of the DMS console and choose All Features > Solution > Data Archiving.

3. In the upper-right corner of the page, click Data Archiving.

4. On the Create Data Archiving Ticket page, configure the following parameters:

   Configuration Item	Required	Description
   Archiving Destination	Yes	Select Dedicated Storage.
   Source Database	Yes	Search for and select the source database.
   Archiving Configuration	Yes	Select the tables whose data you want to archive. Optional: Enter a filter condition for the data to be archived. Example: gmt_modified<='${6_month_ago}'. Note To use a time variable, for example, to archive data from six months ago, configure the variable in the Variable Configuration section first. Optional: Click Add to add multiple source tables.
   Archive Table Mapping	No	In the Archive Table Mapping section, you can view information about the archived tables, such as table names, columns, and database shard keys.
   Variable Configuration	No	Configure variables that you can use in the filter condition. For example, set the time variable to 6_month_ago, the time format to yyyy-MM-dd, and the offset to -6 months. If the current date is August 12, 2021, the value of the ${6_month_ago} variable is 2021-02-11. For more information about how to configure time variables, see Configure time variables.
   Post-behavior	No	Specify whether to delete the archived data from the source tables. If you select Clean up the archived data of the original table (delete-No Lock), the archived data is automatically deleted from the source tables. You can execute the DELETE statement to delete temporary backup tables. The temporary backup tables are generated in the source database to store the archived data when the source tables are deleted. Make sure that the source database has sufficient storage space to prevent instance unavailability caused by insufficient storage space. After the data is archived and you verify that the archived data is correct, you can create a Normal data change ticket to clear the temporary backup tables from the source database. If you do not select Clean up the archived data of the original table (delete-No Lock), the archived data is not deleted from the source tables. In this case, you need to manually delete data from the source tables and optimize the storage usage. To delete the archived data from the source tables, create a regular data change ticket. For more information, see Normal data change. To optimize the storage usage of the source tables, create a lock-free change ticket. For more information, see Perform lockless schema changes using lockless change tickets.
   Run Mode	Yes	Select a run mode. The following modes are supported: One-time Execution: After the ticket is approved, the archiving task runs only once. Recurring Schedule: After the ticket is approved, the archiving task runs based on the configured schedule. For more information, see Recurring schedule.

5. Click Submit.

6. After the ticket is approved, the system automatically runs the data archiving task. Wait for the task to complete.

   If the task fails, go to the execution section of the ticket and click Details in the Actions column to view the task logs and identify the cause of the failure. If the failure is caused by a transient issue, such as a network or database connection failure, you can click Retry from Breakpoint to run the task again.

   Note

   The archived data is saved in the Parquet file format and is compressed.

7. Query data archived to dedicated storage.

Restore archived data to the source database

Delete archived data from dedicated storage

You can permanently delete data that was archived by a one-time or recurring archiving ticket from dedicated storage. This action releases the storage space that is occupied by the data.

FAQ

Q: When data from a logical database is archived, are all tables in the logical database merged into a single physical table?

A: Yes, they are. When data from a logical database is archived, all of its tables are merged into a single physical table. Therefore, on the archive view page, you can query any logical table to view the data of all tables in the logical database.

Q: If a ticket is used for multiple archiving tasks, such as recurring archiving, can I query the data archived within a specific time period?

A: Yes, you can. For a recurring archiving task within the same ticket, archived data for different time periods is generated based on the archiving time. You can use the creation time of the data archive to identify and query data that was archived within a specific time period.

Q: How do I query all data that is periodically archived from a table or all data that is archived from the same table across multiple tickets?

Q: When archiving data from an RDS database to dedicated storage, if I select the option to clean up source table data, how much space in the source database will the temporary backup table occupy?

A: The temporary table data can occupy an amount of space equal to the used space of the source database.