Data partitioning

Partitioned tables

In Fluss, a partitioned table organizes data based on one or more partition keys. This effectively improves query performance and manageability for large datasets. Partitions divide data into segments, each corresponding to a specific partition key value.

Fluss supports the following three management strategies for partitioned tables:

• Manual partition management: You can manually create new partitions or delete existing ones. For more information, see Manage Partitions.

• Automatic partition management: The system automatically creates partitions based on the automatic partitioning rules configured when you create the table. It also automatically cleans up expired partitions to prevent unlimited data growth. For more information, see Automatic Partitioning.

• Dynamic partition creation: Fluss dynamically creates partitions based on the data written to the table. For more information, see Dynamic Partitioning.

These three strategies are orthogonal and can coexist on the same table.

Multi-field partitioned tables

Partitioned tables, including both primary key tables and log tables, support configuring partition keys based on multiple fields. This approach allows you to partition data by combinations of field values, enabling more granular data organization, management, and query optimization.

For example, in an Order primary key table, you can define the partition key as (date, region). Data is then stored in partitions corresponding to specific combinations, such as date=2025-04-05, region='US'. In streaming queries, you can filter data, for example, by region='US', to leverage partition pruning and improve read performance through partition pushdown.

Key advantages of partitioned tables

• Improved query performance: Querying specific partitions reduces the amount of data the system reads, which shortens query execution time.

• Clearer data organization: Partitions help organize data logically, making it easier to manage and query.

• Better scalability: Dividing large datasets into smaller, more manageable chunks improves system scalability.

Limitations

• The partition key type must be STRING.

• Tables with a single partition key support both automatic partition creation and cleanup. Tables with multiple partition keys only support automatic cleanup of expired partitions.

• If the table is a primary key table, the partition key must be a subset of the primary key.

• You can configure automatic partitioning rules only at table creation; they cannot be modified afterward.

Automatic partitioning

CREATE TABLE site_access(
  event_day STRING,  -- Partition key
  site_id INT,       -- Site ID
  city_code STRING,  -- City code
  user_name STRING,  -- User name
  pv BIGINT,         -- Page views
  PRIMARY KEY(event_day, site_id) NOT ENFORCED 
) PARTITIONED BY (event_day) WITH (
  'table.auto-partition.enabled' = 'true',            -- Enable automatic partitioning
  'table.auto-partition.time-unit' = 'YEAR',          -- Set the time granularity to YEAR
  'table.auto-partition.num-precreate' = '5',         -- Pre-create 5 partitions
  'table.auto-partition.num-retention' = '2',         -- Retain 2 historical partitions
  'table.auto-partition.time-zone' = 'Asia/Shanghai'  -- Set the time zone to Asia/Shanghai
);

Dynamic partitioning

Dynamic partitioning is a client-side feature, enabled by default, that automatically creates partitions as data is written to a table. This is valuable when the full set of partitions is unknown in advance, eliminating the need for manual creation. It is also particularly useful when you work with multi-field partitions because automatic partitioning currently supports partition creation for single-field keys only.

Client options

Dynamic partitioning example

Step 1: Create a sample table

In the left navigation bar, choose Data Development > Data Query, create a new, blank data query script, and enter the following SQL to create a partitioned table with the dt column as the partition key.

CREATE TABLE IF NOT EXISTS `fluss-demo`.fluss.table_enable_dynamic_partition
(
    id    BIGINT
    ,name STRING
    ,dt   STRING
)
PARTITIONED BY (dt)
WITH (
    'client.writer.dynamic-create-partition.enabled' = 'true'
)
;

Verify the current partitions of the table:

SHOW PARTITIONS `fluss-demo`.fluss.table_enable_dynamic_partition;

The query result indicates that the table's partition list is empty.

Step 2: Dynamically insert data

Run the following SQL statement to insert data into a non-existent partition:

INSERT INTO `fluss-demo`.fluss.table_enable_dynamic_partition values(1, 'hello', '20250701');

Step 3: Verify partition creation

Run the SHOW PARTITIONS command again to verify that the partition has been dynamically created.

The query result shows dt=20250701 in the partition name column, confirming that the partition was dynamically created.