Archive data to Lindorm

Updated at:

Data Management (DMS) lets you archive historical records from relational databases to a Lindorm instance, reducing storage costs on production databases while keeping data accessible for queries.

Prerequisites

  • The following types of source databases are supported:

    • MySQL: ApsaraDB RDS for MySQL, PolarDB for MySQL, and AnalyticDB for MySQL V3.0

    • PostgreSQL: ApsaraDB RDS for PostgreSQL and PolarDB for PostgreSQL

    • PolarDB-X 2.0

    Note

    The account of a MySQL database must have the REPLICATION CLIENT permission.

  • A Lindorm wide-table engine of version 2.2.17 or later is purchased. For information about the purchase procedure, see Create an instance.

  • The destination Lindorm database is added to DMS. Select Lindorm-SQL when you add the database. For more information, see Register a cloud database instance.

  • Source tables have a primary key or a unique key.

    Note

    We recommend that the source table contains a modification time field so that you can use the business modification time as a filter condition for archiving.

Usage notes

The control modes of both the source database and the destination database must be Security Collaboration or Stable Change to run periodic archive tasks. If you need to run only one-time archive tasks, there are no restrictions on the control mode.

Note

If your instances are in Stable Change control mode, enable security hosting or change the control mode to Security Collaboration before you configure a recurring data archiving task. This prevents task failures that are caused by instance logon expiration. For more information, see Enable security hosting and Change the control mode of an instance.

Billing

The fee for purchasing the destination instance.

Archive data

  1. Log in to DMS 5.0.

  2. In the top navigation bar, choose Solution > Data Archiving.

    Note

    If you use the DMS console in simple mode, move the pointer over the 2023-01-28_15-57-17.png icon in the upper-left corner of the DMS console and choose All Features > Solution > Data Archiving.

  3. In the upper-right corner of the page, click Data Archiving.

  4. On the New Data Archive Ticket page, configure the ticket information.

    Parameter

    Required

    Description

    Task Name

    Yes

    Enter a clear and specific task name to reduce communication costs.

    Archive Destination

    Yes

    Select Lindorm.

    Lindorm Instance

    Yes

    Search for and select the destination ApsaraDB for Lindorm instance.

    Source Database

    Yes

    Search for and select the source database.

    Archive Configuration

    Yes

    Select the table to archive.

    • Optional: Enter the filter conditions for the data to be archived. Example: gmt_modified<='${6_month_ago}'.

      Note

      If you need to use a time variable, for example, to archive data from six months ago, first configure the variable in the variable configuration section.

    • Optional: Click Add to add multiple source tables.

    Archive Table Mapping

    No

    You can click Edit in the Actions column of the table name to edit the table name, column information, sharding key, and partition key after the source table is archived to the destination database.

    Variable Configuration

    No

    Define time variables for filter conditions. For example, create a variable named 6_month_ago with format yyyy-MM-dd and offset -6 Month. If today is August 12, 2021, ${6_month_ago} resolves to 2021-02-11. For more information, see the "Configure time variables" section of the Variables topic.

    Post-action

    No

    • If you select Clean up archived data in the source table (delete-lock-free), after archiving is completed, the system automatically deletes the archived data in the source table by using a lock-free data change DELETE statement. However, deleting the source table generates a temporary backup table in the source database to store the archived data. Ensure that the source database has enough space to avoid instance unavailability caused by insufficient space.

      Clean up the temporary backup table: After you confirm that data archiving is complete and the data is correct, you can create a Standard data change ticket to clean up the temporary backup table in the source database.

    • If you do not select Clean up archived data in the source table (delete-lock-free), you need to manually delete data in the source table and optimize space after data archiving is complete.

      1. Clean up data in the source table: Create a normal data change ticket to clean up the archived data in the database. For information about how to create a normal data change ticket, see Standard data change.

      2. Optimize space in the source table: Create a lock-free change ticket to optimize the space of the source table. For information about how to create a lock-free change ticket, see Perform lockless schema changes using lockless change tickets.

    • If you select Verify archived data, the archived data is verified when the archive task is run. The verification includes the archive table name, source database, and the number of SQL statements. You can view the details by clicking Verification Details.

    Run Mode

    Yes

    Select a run mode. The following modes are supported:

    • Run Once: After the ticket is approved, the archive task runs only once.

    • Periodic scheduling: After the ticket is approved, the archive task runs based on the deployed periodic schedule. For more information, see Periodic scheduling.

  5. Click Submit.

  6. After the ticket is approved, DMS automatically runs the data archiving task.

  7. Query the data after archiving is successful.

Query archived data

Method 1: Query archived data by using DMS

  1. After the data archive task runs successfully, on the current Ticket Details page, in the Basic Information section, click View on the right side of Destination Database.

  2. After you enter the SQL Console page, in the table list on the left, select and double-click the destination table name, and then click Execute to view the archived data.

    Note

    The system automatically creates databases and tables in the archive instance based on the database and table names in the source database. Therefore, the destination database name is the same as the source database name.

    Four new columns are added to the archived table without affecting the use of the original data:

    • Archive information (ticket ID and archive time)

    • Database name

    • Table name

    • Instance ID (the ID assigned when the instance is added to DMS, corresponding to the actual instance ID)

Method 2: Query archived data by using Lindorm

Wide-table engine Data query.

Periodic scheduling

Parameter

Description

Scheduling Cycle

Select the cycle for scheduling the task:

  • Hour: Runs the task based on the specified hour. You need to configure a scheduled schedule.

  • Day: Runs the task once a day. You need to configure a specific time for daily scheduling.

  • Week: Runs the task on a weekly cycle, on each specified day. You need to configure the specified time and specific time.

  • Month: Runs the task on a monthly cycle, on each specified day. You need to configure the specified time and specific time.

Scheduled Schedule

The following two scheduled scheduling methods are provided:

  • Fixed interval scheduling:

    • Start Time: The start time for running the task.

    • Interval: The interval for running the task. Unit: hours.

    • End Time: The end time for running the task.

    For example, if you set Start Time to 00:00, Interval to 6 hours, and End Time to 20:59, the system runs the task at 0, 6, 12, and 18 o'clock.

  • Specified Time scheduling: Select the target time points for running the task.

    For example, if you select 0 hours and 5 hours, the system runs the task at 0 and 5 o'clock.

Specified Time

  • If the scheduling cycle is Week, select the days of the week to run the task. Multiple selections are supported.

  • If the scheduling cycle is Month, select the days of the month to run the task. Multiple selections are supported.

Specific Time

Set the specific time to run the task flow.

For example, if you set the time to 02:55, the system runs the task at 02:55 on the specified day.

Cron Expression

You do not need to configure this manually. The system automatically displays it based on the scheduling cycle and specific time that you configure.