Incremental clustering

Use cases

Incremental Clustering delivers the most benefit in the following situations:

• Continuous data ingestion with filter queries: The table receives a steady stream of new data, and queries frequently filter on specific columns such as event_time or user_id. Each clustering run tightens the data layout so filters become more selective over time.

• High-cardinality filter columns: Queries filter on columns with many distinct values (for example, user IDs or device IDs). Without clustering, the query engine must scan many files to find matching rows. After clustering, files are physically sorted so the engine can skip most of them.

• Accumulating small files: The ingestion pipeline writes many small files. Incremental Clustering merges these into files close to the target-file-size in a single pass, reducing file count and improving read throughput.

• Historical partitions with stale layout: Partitions that have stopped receiving new data still have a poorly sorted layout from early ingestion. Enabling full clustering for historical partitions reorganizes them when resources allow, without affecting active ingestion.

How it works

Once enabled, DLF automatically schedules Incremental Clustering runs. Each run does the following in one pass:

1. Selects and rewrites data files. During each run, the system selects a subset of data files for rewriting. The rewritten files have a more compact statistical range for the specified clustering columns, meaning a narrower spread between minimum and maximum values. This allows the query engine to skip irrelevant data files and significantly reduces the amount of data scanned. Larger datasets see a greater performance improvement.

2. Merges small files. The rewrite follows the target-file-size setting, so multiple small files are merged into larger files as part of the same operation.

Partitions that have been idle longer than a configurable threshold are promoted to full clustering, which reorganizes all files in the partition rather than just new arrivals. Full clustering runs at lower priority than incremental runs and only when sufficient resources are available.

Prerequisites

Before you begin, ensure that:

• The table is a Paimon append-only table

• row-tracking.enabled is not set to true

• data-evolution.enabled is not set to true

• deletion-vectors.enabled is not set to true for Append Bucketed tables

Enable Incremental Clustering

The DLF console does not currently provide a UI for this feature. Configure it by running ALTER TABLE with the following table properties:

For append-only tables with bucket > 0 (Append Bucketed tables), you must also disable the default write-order guarantee by setting the following additional parameter:

Example 1 (for Paimon append-only tables with bucket = -1):

ALTER TABLE catalog.db.my_table SET (
  'clustering.incremental' = 'true',
  'clustering.columns' = 'event_time,user_id',
  'clustering.strategy' = 'zorder'
);

Example 2 (for Paimon append-only tables with bucket > 0):

ALTER TABLE catalog.db.my_table SET (
  'bucket-append-ordered' = 'false',
  'clustering.incremental' = 'true',
  'clustering.columns' = 'event_time,user_id',
  'clustering.strategy' = 'zorder'
);

Adjust the scheduling interval

By default, DLF schedules an Incremental Clustering run once per hour. Shorten the interval to optimize freshly ingested data faster; increase it to reduce system load.

Example:

ALTER TABLE catalog.db.my_table SET ('morax.compact.check-interval' = '30min');

Enable full clustering for historical partitions (optional)

A partition is considered historical when it has not received new data for longer than the threshold you configure. Full clustering reorganizes all files in historical partitions, not just those added since the last run.

Example:

ALTER TABLE catalog.db.my_table SET (
  'clustering.history-partition.idle-to-full-sort' = '3d',
  'clustering.history-partition.limit' = '3'
);

Verify clustering status

Paimon data files have a level property:

• Newly written files start at level = 0.

• Files that have been through a clustering run are promoted to level 1 through level 5.

Query the $files metadata table to confirm clustering has run on a partition:

SELECT *
FROM `catalog_name`.`database_name`.`table_name$files`
WHERE `partition` = 'your_partition_value';

If the result set contains rows where level > 0, clustering has been applied to that partition. To measure the performance improvement, compare query execution times or inspect query plans before and after enabling clustering.