Archive data to Lindorm
Data Management (DMS) lets you archive historical records from relational databases to a Lindorm instance, reducing storage costs on production databases while keeping data accessible for queries.
Prerequisites
The following types of source databases are supported:
-
MySQL: ApsaraDB RDS for MySQL, PolarDB for MySQL, and AnalyticDB for MySQL V3.0
-
PostgreSQL: ApsaraDB RDS for PostgreSQL and PolarDB for PostgreSQL
-
PolarDB-X 2.0
NoteThe account of a MySQL database must have the REPLICATION CLIENT permission.
-
A Lindorm wide-table engine of version 2.2.17 or later is purchased. For information about the purchase procedure, see Create an instance.
The destination Lindorm database is added to DMS. Select Lindorm-SQL when you add the database. For more information, see Register a cloud database instance.
Source tables have a primary key or a unique key.
NoteWe recommend that the source table contains a modification time field so that you can use the business modification time as a filter condition for archiving.
Usage notes
The control modes of both the source database and the destination database must be Security Collaboration or Stable Change to run periodic archive tasks. If you need to run only one-time archive tasks, there are no restrictions on the control mode.
If your instances are in Stable Change control mode, enable security hosting or change the control mode to Security Collaboration before you configure a recurring data archiving task. This prevents task failures that are caused by instance logon expiration. For more information, see Enable security hosting and Change the control mode of an instance.
Billing
The fee for purchasing the destination instance.
Archive data
Log in to DMS 5.0.
-
In the top navigation bar, choose .
NoteIf you use the DMS console in simple mode, move the pointer over the
icon in the upper-left corner of the DMS console and choose . In the upper-right corner of the page, click Data Archiving.
On the New Data Archive Ticket page, configure the ticket information.
Parameter
Required
Description
Task Name
Yes
Enter a clear and specific task name to reduce communication costs.
Archive Destination
Yes
Select Lindorm.
Lindorm Instance
Yes
Search for and select the destination ApsaraDB for Lindorm instance.
Source Database
Yes
Search for and select the source database.
Archive Configuration
Yes
Select the table to archive.
Optional: Enter the filter conditions for the data to be archived. Example:
gmt_modified<='${6_month_ago}'.NoteIf you need to use a time variable, for example, to archive data from six months ago, first configure the variable in the variable configuration section.
Optional: Click Add to add multiple source tables.
Archive Table Mapping
No
You can click Edit in the Actions column of the table name to edit the table name, column information, sharding key, and partition key after the source table is archived to the destination database.
Variable Configuration
No
Define time variables for filter conditions. For example, create a variable named
6_month_agowith formatyyyy-MM-ddand offset-6 Month. If today is August 12, 2021,${6_month_ago}resolves to2021-02-11. For more information, see the "Configure time variables" section of the Variables topic.Post-action
No
If you select Clean up archived data in the source table (delete-lock-free), after archiving is completed, the system automatically deletes the archived data in the source table by using a lock-free data change
DELETEstatement. However, deleting the source table generates a temporary backup table in the source database to store the archived data. Ensure that the source database has enough space to avoid instance unavailability caused by insufficient space.Clean up the temporary backup table: After you confirm that data archiving is complete and the data is correct, you can create a Standard data change ticket to clean up the temporary backup table in the source database.
If you do not select Clean up archived data in the source table (delete-lock-free), you need to manually delete data in the source table and optimize space after data archiving is complete.
Clean up data in the source table: Create a normal data change ticket to clean up the archived data in the database. For information about how to create a normal data change ticket, see Standard data change.
Optimize space in the source table: Create a lock-free change ticket to optimize the space of the source table. For information about how to create a lock-free change ticket, see Perform lockless schema changes using lockless change tickets.
If you select Verify archived data, the archived data is verified when the archive task is run. The verification includes the archive table name, source database, and the number of SQL statements. You can view the details by clicking Verification Details.
Run Mode
Yes
Select a run mode. The following modes are supported:
Run Once: After the ticket is approved, the archive task runs only once.
Periodic scheduling: After the ticket is approved, the archive task runs based on the deployed periodic schedule. For more information, see Periodic scheduling.
Click Submit.
After the ticket is approved, DMS automatically runs the data archiving task.
Query archived data
Method 1: Query archived data by using DMS
After the data archive task runs successfully, on the current Ticket Details page, in the Basic Information section, click View on the right side of Destination Database.
After you enter the SQL Console page, in the table list on the left, select and double-click the destination table name, and then click Execute to view the archived data.
NoteThe system automatically creates databases and tables in the archive instance based on the database and table names in the source database. Therefore, the destination database name is the same as the source database name.
Four new columns are added to the archived table without affecting the use of the original data:
Archive information (ticket ID and archive time)
Database name
Table name
Instance ID (the ID assigned when the instance is added to DMS, corresponding to the actual instance ID)
Method 2: Query archived data by using Lindorm
Wide-table engine Data query.
Periodic scheduling
Parameter | Description |
Scheduling Cycle | Select the cycle for scheduling the task:
|
Scheduled Schedule | The following two scheduled scheduling methods are provided:
|
Specified Time |
|
Specific Time | Set the specific time to run the task flow. For example, if you set the time to 02:55, the system runs the task at 02:55 on the specified day. |
Cron Expression | You do not need to configure this manually. The system automatically displays it based on the scheduling cycle and specific time that you configure. |