Table features

Overview

• remote storage: To ensure low latency, the most recent data is retained on local disks. Older data is asynchronously migrated to remote storage to reduce local storage costs.

• column pruning: During a query, only required columns are read and irrelevant ones are skipped, which reduces the amount of data read.

• predicate pushdown: The server uses column statistics to skip data blocks that do not match filters. This reduces network I/O and deserialization overhead.

• log compression: This feature uses the native Arrow compression mechanism to significantly reduce network transfer and storage costs.

Remote storage

Remote storage refers to a more cost-effective and reliable external storage system, such as Amazon S3, Hadoop Distributed File System (HDFS), or Alibaba Cloud OSS. By default, Fluss uses Alibaba Cloud OSS as its remote storage.

• Fluss uses remote storage for data tiering and fault tolerance.

• For log tables, remote storage stores tiered log segments.

• For primary key tables, remote storage stores snapshots and change log data segments.

Column pruning

Column pruning is an optimization that reduces the amount of data read by only reading the required columns and skipping irrelevant ones during a query. Fluss supports column pruning for stream reads, making it one of the few systems in the industry with this capability for real-time stream processing scenarios.

• Fluss supports column pruning for both log tables and the change logs of primary key tables. This feature reduces storage I/O, decreases network traffic, and lessens the parsing load on the client, significantly improving query performance.

• For both batch processing and stream processing, the system always processes only the necessary fields, which maximizes resource utilization.

When a compute engine, such as Flink, executes a query, it analyzes the required columns and instructs Fluss to return only those columns.

SELECT id, name FROM log_table WHERE timestamp > '2023-01-01'

This query reads only the id, name, and timestamp columns. Fluss does not read other columns in the table, such as address and status, from storage, which avoids unnecessary reads.

Predicate pushdown

Predicate pushdown is a server-side optimization that applies only to log tables. The server evaluates filter predicates against the column statistics (min, max, and nullCount) of each RecordBatch. It then skips entire batches that cannot contain matching rows, which reduces network I/O and deserialization overhead.

CREATE TABLE log_table (
  order_id BIGINT,
  item_id BIGINT,
  amount INT,
  address STRING
) WITH (
  'table.statistics.columns' = 'order_id,amount'
);

ALTER TABLE log_table SET ('table.statistics.columns' = 'order_id,amount');

Log compression

Fluss uses the native Arrow compression mechanism. The compressed data remains compliant with the Arrow format standard. Fluss compresses each column independently, which allows column pruning to function and does not affect query performance.

Compression process

• For log tables, the writer on the client side compresses data and writes it to storage in a compressed format. When reading the data, the log scanner decompresses it.

• For the change logs of primary key tables, the server generates and compresses the log.

Benefits of compression

Compression significantly reduces network transfer and storage costs. Benchmark tests show that ZSTD at compression level 3 can achieve a compression ratio of approximately 5:1 (for example, compressing 5 GB of data down to 1 GB). The reduction in network overhead also increases read and write throughput.

Configuration

Configuration example

-- Set the compression algorithm to ZSTD and the compression level to 2
CREATE TABLE log_table (
  order_id BIGINT,
  item_id BIGINT,
  amount INT,
  address STRING
) WITH (
  'table.log.arrow.compression.type' = 'ZSTD',
  'table.log.arrow.compression.zstd.level' = '2'
);