This topic describes how to define, create, modify, and delete Time to Live (TTL) tables.
Scope
Row-based archiving
For engine version MySQL 5.7, the instance version must be polardb-2.4.0_5.4.19-20240927_xcluster5.4.19-20240920 or later.
For engine version MySQL 8.0, the instance version must be polardb-2.4.0_5.4.19-20240927_xcluster8.4.19-20240924 or later.
Partition-based archiving
For engine version MySQL 5.7, the instance version must be polardb-2.5.0_5.4.20-20250328_xcluster5.4.20-20250221 or later.
For engine version MySQL 8.0, the instance version must be polardb-2.5.0_5.4.20-20250328_xcluster8.4.20-20250304 or later.
-
For information about the instance version naming rules, see Release notes.
-
For information about how to view the version of an instance, see View and update the engine version of a PolarDB for Xscale instance.
Add a TTL definition
In PolarDB-X, you can use the ALTER statement to specify a TTL definition for a table, which defines the data expiration rules for that table. A table with a TTL definition is called a TTL table.
Adding a TTL definition only updates the TTL metadata of an online table. It does not trigger any data cleanup or partition changes and has no impact on online workloads.
TTL definitions are supported only for partitioned tables in AUTO mode databases. Tables that use
Local Partitionare not supported.
Syntax
ALTER TABLE table_name
MODIFY TTL
SET
TTL_EXPR = xxx
[,TTL_JOB = xxx]
[,TTL_PART_INTERVAL = xxx]
[,TTL_FILTER = xxx]
[,TTL_ENABLE = xxx]
[,TTL_CLEANUP = xxx]
[,ARCHIVE_TYPE = xxx]
[,ARCHIVE_TABLE_PRE_ALLOCATE = xxx]
[,ARCHIVE_TABLE_POST_ALLOCATE = xxx]
;For detailed descriptions of each parameter, see TTL definition parameters.
Example
Create an example partitioned table named my_ttl_tbl:
CREATE TABLE `my_ttl_tbl` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`date_field` datetime DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4
PARTITION BY KEY(`id`)
PARTITIONS 8;Add a TTL definition:
ALTER TABLE my_ttl_tbl
MODIFY TTL
SET
TTL_EXPR = `date_field` EXPIRE AFTER 2 MONTH TIMEZONE '+08:00';TTL definition parameters
A TTL definition in PolarDB-X 2.0 consists of several parts: TTL_EXPR, TTL_JOB (optional), TTL_PART_INTERVAL (optional), and TTL_FILTER (optional).
TTL_EXPR
Defines the TTL time column, data retention period, and data expiration policy.
Syntax
TTL_EXPR = data_expiration_expr_definition data_expiration_expr_definition: data_expire_by_time_defition | data_expire_by_part_count_defition data_expire_by_time_defition: ttl_time_column_expr EXPIRE AFTER data_expire_expr_definition timezone_definition ttl_time_column_expr: ttl_time_col | FROM_UNIXTIME(ttl_ts_int_col) | FROM_UNIXTIME(ttl_milli_ts_int_col/1000) data_expire_expr_definition: EXPIRE AFTER ttl_expire_interval_definition ttl_expire_interval_definition: expire_after_interval_count interval_timeunit_expr interval_timeunit_expr: DAY | MONTH | YEAR timezone_definition: TIMEZONE time_zone_value time_zone_value: '[+-]HH:MM' expire_after_interval_count: int_number data_expire_by_part_count_defition: ttl_int_col data_expire_over_definition data_expire_over_definition: EXPIRE OVER expire_over_partition_count PARTITIONS expire_over_partition_count: int_numberNoteThe
data_expiration_expr_definitionparameter supports the following two data expiration policies:data_expire_by_time_defition: Data expires automatically after a fixed time interval. Data stored in a TTL table for longer than the specified interval is considered expired. This policy is typically used for time-based archiving and is referred to as the EXPIRE AFTER policy in this topic.ttl_time_col: The TTL column. The following data types are supported:Time types: DATE, DATETIME, and TIMESTAMP.
Integer types: INT and BIGINT.
ttl_ts_int_col: A column of an integer type that represents a timestamp in seconds, such as a UNIX timestamp. The expressionFROM_UNIXTIME(ttl_ts_int_col)converts the integer timestamp to a time value for expiration checks.ttl_milli_ts_int_col: A column of an integer type that represents a timestamp in milliseconds, such as a Java timestamp. The expressionFROM_UNIXTIME(ttl_ts_int_col/1000)converts the millisecond timestamp to a time value for expiration checks.ttl_expire_interval_definition: The time-to-live (TTL) for data, in the format
IntervalNumber IntervalTimeUnit. When(now() - ttl_time_column_expr) > ttl_expire_interval_definition, data that meets this condition is automatically deleted. The supported values forIntervalTimeUnitareDAY/MONTH/YEAR. Example:TTL_EXPR = `ttl_col` EXPIRE AFTER 3/*intervalNumber*/ MONTH /*intervalTimeUnit*/ TIMEZONE '+08:00'The expiration interval number
IntervalNumberis 3, and the expiration interval unitIntervalTimeUnitisMONTH.The expiration interval unit
IntervalTimeUnitdirectly affects the truncation and normalization of the upper boundary of the cleanup time range. For more information, see TTL table cold data cleanup algorithm.IntervalTimeUnit is the default time unit for the
TTL_PART_INTERVALexpression. IfTTL_PART_INTERVALis not explicitly set, the system usesIntervalTimeUnitas the partition interval unit.
timezone_definition: Defines the time zone of the TTL column. The scheduled TTL task uses the time zone to calculate expiration. You must specify a time zone if the TTL column is of the TIMESTAMP type. Only the'[+-]HH:MM'format is supported.
data_expire_by_part_count_defition: Data expires automatically based on a fixed number of RANGE partitions. The TTL table retains a maximum ofexpire_over_partition_countpartitions and deletes the oldest RANGE partition. This policy is typically used for partition-based archiving and is referred to as the EXPIRE OVER policy in this topic. This policy is often used with theTTL_PART_INTERVALattribute.ttl_int_col: A TTL column of an integer type.expire_over_partition_count: The maximum number of partitions to retain.
Examples
The TTL column is of type DATETIME. The TTL table retains data from the last three months.
TTL_EXPR = `<ttl_col>` EXPIRE AFTER 3 MONTH TIMEZONE '+08:00'The TTL column is of type TIMESTAMP and uses the UTC+10 time zone. The TTL table retains data from the last 30 days.
TTL_EXPR = `<ttl_col>` EXPIRE AFTER 30 DAY TIMEZONE '+10:00'The TTL column is of type INT and represents a UNIX timestamp in seconds. The TTL column uses the UTC+0 time zone. The TTL table retains data from the last 7 days.
TTL_EXPR = FROM_UNIXTIME(`<ttl_col>`) EXPIRE AFTER 7 DAY TIMEZONE '+00:00'The TTL column is of type INT and represents a timestamp in milliseconds. The TTL column uses the UTC+8 time zone. The TTL table retains data from the last 60 days.
TTL_EXPR = FROM_UNIXTIME(`<ttl_col>`/1000) EXPIRE AFTER 60 DAY TIMEZONE '+08:00'The TTL column is of type BIGINT and represents an auto-incrementing business serial number. The TTL table retains a maximum of 16 RANGE partitions.
TTL_EXPR = `<ttl_col>` EXPIRE OVER 16 PARTITIONS
TTL_JOB
Defines the submission time and frequency of the scheduled TTL task. This attribute is optional. The default value is '0 0 1 */1 * ? *', which runs the task once a day at 01:00.
Syntax
TTL_JOB = quartz_cron_expr_definition[timezone_definition] quartz_cron_expr_definition : CRON cron_expr_value timezone_definition : TIMEZONE time_zone_value time_zone_value: '[+-]HH:MM'Parameters
Parameter
Description
cron_expr_value
Specifies the time and frequency of the system cleanup task by using a Quartz CRON expression. The expression has seven fields:
Second (0-59)
Minute (0-59)
Hour (0-23, use
*for every hour)Day of month (1-31. The asterisk
*specifies every day.)Month (1-12 or JAN, FEB, MAR, APR, ..., DEC, where
*represents every month)Day of week (1-7 or SUN, MON, TUE, WED, THU, FRI, SAT, where 1 is Sunday. Use ? as a placeholder if this field is not set.)
Year (use
*to represent every year)
timezone_definition (optional)
The
time_zone_valueintimezone_definitionis fixed to'+08:00'. By default, TTL tasks are scheduled based on the UTC+8 time zone. Other time zones are not supported.ImportantBecause scheduled cleanup tasks consume system resources, PolarDB-X 2.0 does not support setting the task frequency to the minute or second level. We also recommend avoiding an hourly frequency to prevent frequent task execution from affecting service stability.
Example
The TTL background cleanup task is run once a day at 01:00:
TTL_JOB = CRON '0 0 1 */1 * ? *' TIMEZONE '+08:00'Notes
TTL_JOBdefines the time and frequency at which the scheduled TTL task is automatically submitted. However, the actual execution of the TTL task is constrained by the maintenance window (the default window is from 02:00 to 05:00).A TTL task that is not completed within the current maintenance window is automatically paused and resumes during the next maintenance window. For example, if you set
TTL_JOB = CRON '0 0 1 */1 * ? *' TIMEZONE '+08:00'and the maintenance window is from 02:00 to 05:00, the TTL task is submitted daily at 01:00 but starts running at 02:00. If the task is not completed by 05:00, it is automatically paused and resumes at 02:00 the next day.Ensure the TTL task can be completed within the maintenance window. Otherwise, residual data may accumulate.
TTL_PART_INTERVAL
Specifies the time or numeric interval between adjacent partitions of a TTL table and its archive table. This attribute applies in the following three main scenarios:
When a TTL table uses partition-based archiving, the scheduled task automatically pre-creates new RANGE partitions for the TTL table. The interval between these new partitions is specified by
TTL_PART_INTERVAL.When an archive table is created to store archived data, the database automatically defines RANGE partitions for the archive table. The time interval for these RANGE partitions is specified by
TTL_PART_INTERVAL.After an archive table is created, the scheduled task automatically pre-creates new RANGE partitions for it. The interval between these new partitions is specified by
TTL_PART_INTERVAL.
This attribute is typically used with TTL_EXPR.
Syntax
TTL_PART_INTERVAL = INTERVAL(int_value, interval_time_unit) interval_time_unit : DAY | MONTH | YEAR | NUMBERNoteIf the interval unit is
NUMBER, the TTL column must be of an integer type and use the EXPIRE OVER expiration policy. The value must be a positive integer.In partition-based archiving, if you specify only the expiration interval in
TTL_EXPR(for example,`ttl_col` EXPIRE AFTER 3 MONTH TIMEZONE '+08:00') and do not explicitly setTTL_PART_INTERVAL, the default value is determined as follows:The time interval unit of
TTL_PART_INTERVALis the same as the expiration interval unit ofTTL_EXPR.The default interval number of
TTL_PART_INTERVALis 1.
Examples
Partition by year (one partition per year).
TTL_PART_INTERVAL = INTERVAL(1, YEAR)Partition by month (one partition per month).
TTL_PART_INTERVAL = INTERVAL(1, MONTH)Partition by week (one partition every 7 days).
TTL_PART_INTERVAL = INTERVAL(7, DAY)Partition by day (one partition per day).
TTL_PART_INTERVAL = INTERVAL(1, DAY)Partition by integer range (one partition per interval of 100,000).
TTL_PART_INTERVAL = INTERVAL(100000, NUMBER)
TTL_FILTER
For TTL tables that use row-based archiving, use this attribute to specify additional filtering conditions. This allows the TTL task to clean up only the expired data that meets specific business criteria.
Syntax
TTL_FILTER = COND_EXPR (query_cond_expr)query_cond_exprrefers to the conditional expression in the WHERE clause of a standard DELETE statement. The default value is''. This expression is combined with the time condition for TTL row-level cleanup by using theANDoperator. For example, if the current cleanup time range isttl_col < '2025-05-01'andquery_cond_exprisstatus=1, the actual cleanup filter condition is(( ttl_col < '2025-05-01' ) AND ( status = 1)). Because the conditional expression forquery_cond_exprcannot use the index of the TTL column, you should keep the expression as simple as possible. A complex expression may affect the row-level cleanup speed.ImportantBy default, the cleanup task treats data rows with a NULL value in the TTL column as having the minimum value and cleans them up. You can set
TTL_FILTER = COND_EXPR( <ttl_col_name> IS NOT NULL ))to prevent the cleanup task from automatically cleaning up data rows with a NULL value in the TTL column. This setting is supported only on instance versionspolardb-2.5.0_5.4.20-20250328_xcluster5.4.20-20250221(MySQL 5.7) and later orpolardb-2.5.0_5.4.20-20250328_xcluster8.4.20-20250304(MySQL 8.0) and later.Example
The data cleanup rule applies only to expired data that is marked as completed (status=1):
TTL_FILTER = COND_EXPR(`status` = 1)
Other parameters
Parameter | Description | Example |
TTL_ENABLE | Enables or disables the scheduled TTL task. Valid values:
|
|
TTL_CLEANUP | Enables or disables the cleanup of expired data in the TTL table. Valid values:
Note
|
|
ARCHIVE_TYPE (required) | Specifies the archiving type for the TTL table. Valid values:
|
|
ARCHIVE_TABLE_PRE_ALLOCATE | The number of |
|
ARCHIVE_TABLE_POST_ALLOCATE | When a new archive table is created, this parameter specifies the number of |
|
TTL definition examples
The TTL feature of PolarDB-X currently supports the following two archiving methods:
Row-based archiving: The TTL table uses
DELETEDML statements to clean up expired data. All cleaned-up data is archived to a columnar store in advance.Partition-based archiving: The TTL table uses
DROP (SUB)PARTITIONDDL statements to clean up expired data. All data from the cleaned-up partitions is archived to a columnar store in advance. This method includes two expiration policies:EXPIRE AFTER policy: Data expires after a fixed time interval. Data that has been stored for longer than the specified interval is considered expired.
EXPIRE OVER policy: Data expires based on a fixed number of partitions. When the number of RANGE partitions exceeds the specified limit, the oldest partition is considered expired by default. This policy is typically used for archiving data where the TTL column is of an integer type.
The following sections provide examples of how to use TTL definitions for both archiving methods in various scenarios.
Row-based archiving
The TTL table first archives expired data to OSS by uploading or synchronizing it through a columnar index. Then, it cleans up the expired data by using DELETE DML statements.
In row-based archiving, DELETE statements are used for cleanup. The cleanup logic varies based on the data type of the TTL column, primarily time types and integer types. The following sections provide examples for both.
Partition-based archiving
The TTL table first archives expired data to OSS by uploading or synchronizing it through a columnar index. This step is skipped if no archive table is configured. Then, it cleans up the expired data by using DROP (SUB)PARTITION DDL statements.
Partition-based archiving uses DROP (SUB)PARTITION DDL statements to clean up expired RANGE partitions. The following sections provide examples for archiving by primary partition and by secondary partition.
Note: If an existing online table has a JOIN relationship with other online tables, changing it to a TTL table that uses the partition-based archiving policy may affect the JOIN performance (for example, JOIN pushdown may become ineffective). Therefore, perform a thorough evaluation before making this change.
This is because a TTL table that uses partition-based archiving will automatically drop expired partitions and add new pre-allocated partitions during subsequent scheduled TTL tasks, which changes the table's partition definition. As a result, the partition definitions of this table and the online tables it is joined with may no longer be aligned, which can cause JOIN pushdown to become ineffective.
Archive by primary partition
Expiration by time interval
Expiration by partition count
Archive by secondary partition
Expiration by time interval
Expiration by partition count
Modify a TTL definition
You can use SQL to modify a TTL definition. The changes take effect starting from the second execution cycle after the modification.
Modify task switch (TTL_ENABLE)
By default, TTL_ENABLE is set to OFF. To enable automatic scheduling of the TTL task for the table, run the following SQL statement:
ALTER TABLE `my_ttl_tbl`
MODIFY TTL
SET
TTL_ENABLE = 'ON';The table my_ttl_tbl must have a TTL definition.
Modify data cleanup switch (TTL_CLEANUP)
By default, TTL_CLEANUP is set to OFF. When TTL_CLEANUP = 'OFF', the TTL task does not clean up any data or delete any expired partitions, but it automatically adds new partitions to the TTL table or archive table. Therefore, if you need to create an archive table to store historical data, enable this data cleanup switch only after the archive table is created. This practice prevents data from being cleaned up before it is archived.
ALTER TABLE `my_ttl_tbl`
MODIFY TTL
SET
TTL_CLEANUP = 'ON';Modify data expiration definition (TTL_EXPR)
Assume the TTL_EXPR definition for the table my_ttl_tbl is:
TTL_EXPR = `date_field` EXPIRE AFTER 2 YEAR TIMEZONE '+08:00';To change the expiration period to retain data from the last year, modify the TTL_EXPR definition:
ALTER TABLE `my_ttl_tbl`
MODIFY TTL
SET
TTL_EXPR = `date_field` EXPIRE AFTER 12 MONTH TIMEZONE '+08:00';Modify task schedule (TTL_JOB)
To change the cleanup task schedule to 03:00 every day, run the following statement:
ALTER TABLE `my_ttl_tbl`
MODIFY TTL
SET
TTL_JOB = CRON '0 0 3 */1 * ? *' TIMEZONE '+08:00';The actual execution of the TTL task is also constrained by the maintenance window. All automatically triggered background tasks can run only within the specified daily maintenance window. The default maintenance window is from 02:00 to 05:00 (UTC+8).
Modify partition interval (TTL_PART_INTERVAL)
To change the partition interval to one partition per month (INTERVAL(1,MONTH)), run the following statement:
ALTER TABLE `my_ttl_tbl`
MODIFY TTL
SET
TTL_PART_INTERVAL = INTERVAL(1,MONTH);In partition-based archiving scenarios,
TTL_PART_INTERVALapplies to both the TTL table and its archive table. This is because partitions are automatically and synchronously added to both tables, which ensures that their partitions remain aligned. For example, if the TTL task prepares to pre-allocate a new partition for'2025-06-01'in the TTL table, the same partition for'2025-06-01'is also added to its archive table if one exists.In row-based archiving scenarios,
TTL_PART_INTERVALapplies only to the archive table. This is because the archive table is mandatorily partitioned by range on the TTL column, while the original partitioning scheme of the TTL table remains unchanged.
Modify filter condition (TTL_FILTER)
For row-based archiving, you can dynamically modify the filter condition used for data cleanup. For example, to clean up only data for completed orders (`order_status`=1), run the following statement:
ALTER TABLE `my_ttl_tbl`
MODIFY TTL
SET
TTL_FILTER=COND_EXPR(`order_status`=1);The column used in the filter condition must exist. Otherwise, an error occurs when the cleanup task starts.
You can use an index to optimize query performance. We recommend that you use an indexed column in the filter condition.
We recommend that you do not use complex filter conditions, such as subqueries or large
INlists, to avoid affecting cleanup performance.The filter condition expression and the TTL column expiration condition (
ttl_col < 'yyyy-MM-dd') are always combined with anANDoperator. This meansTTL_FILTERcan only further filter data that is already considered expired.
To clear all additional filter conditions defined by TTL_FILTER, run the following statement:
ALTER TABLE `my_ttl_tbl`
MODIFY TTL
SET
TTL_FILTER=COND_EXPR(TRUE);Modify archiving type (ARCHIVE_TYPE)
You can dynamically change the archiving policy of a TTL table by switching between partition-based and row-based archiving. Run the following SQL statement:
-- Change to row-based archiving
ALTER TABLE `my_ttl_tbl`
MODIFY TTL
SET
ARCHIVE_TYPE = 'ROW';Before you switch from row-based archiving to partition-based archiving, ensure that the TTL table is partitioned by the TTL column. Otherwise, the conversion fails.
Modify pre-allocated partitions (ARCHIVE_TABLE_PRE_ALLOCATE)
To change the number of pre-allocated partitions for the TTL table to 3, run the following statement:
ALTER TABLE `my_ttl_tbl`
MODIFY TTL
SET
ARCHIVE_TABLE_PRE_ALLOCATE = 3;Remove TTL definition
ALTER TABLE `my_ttl_tbl` REMOVE TTL;You cannot directly remove the TTL definition if the TTL table has an archive table. You must delete the archive table first.