Data cache for data lakes

Overview

In data lake analytics, StarRocks frequently scans data files on HDFS or object storage. For ad-hoc queries and other scenarios that repeatedly access the same data, remote I/O can become a performance bottleneck.

Available since v2.5, Data Cache splits remote data into blocks and caches them on local BE nodes. This eliminates redundant remote reads and accelerates hot data queries.

Limitations

• Data Cache for data lakes is supported in StarRocks v2.5 and later, and is enabled by default starting from v3.3.0.

• It applies only to queries on external tables and External Catalogs, not to queries on native tables.

• Supported External Catalog types: those using the StarRocks Native File Reader (Hive, Iceberg, Hudi, Delta Lake, and Paimon). Catalogs that access data via JNI (such as JDBC Catalog) are not supported.

Cache media

StarRocks uses the memory and disks of BE/CN nodes as cache media and supports the following caching modes:

• Memory-only cache: Fastest speed, but limited by memory capacity.

• Two-tier cache (memory + disk): Expands cache capacity, balancing performance and cost.

Cache eviction mechanism

The memory and disk tiers evict data independently. In a two-tier cache, eviction works as follows:

• The system reads from memory first. On a memory miss, it reads from disk and promotes the data into memory.

• Data evicted from memory is written to disk. Data evicted from disk is discarded.

StarRocks Data Cache supports two eviction policies: LRU and SLRU (Segmented LRU). SLRU is the default policy.

SLRU divides cache space into an eviction segment and a probationary segment, both using LRU internally. New data enters the eviction segment. If accessed again, it is promoted to the probationary segment. Data evicted from the probationary segment moves back to the eviction segment; data evicted from the eviction segment is removed from cache. Compared to LRU, SLRU better protects frequently accessed data from being flushed by large one-time scans.

Enable and disable Data Cache

Data Cache is controlled by the system variable enable_scan_datacache and the BE parameter datacache_enable, both enabled by default since v3.3.0. Check the current status:

SHOW VARIABLES LIKE 'enable_scan_datacache';

Key configuration parameters:

To disable Data Cache, run the following command:

SET GLOBAL enable_scan_datacache = false;

Cache population rules

Since v3.3.2, StarRocks applies the following rules to improve cache hit rates:

• Non-SELECT queries (such as ANALYZE TABLE and INSERT INTO SELECT) do not populate the cache.

• Queries that scan all partitions of a table do not populate the cache (unless the table has only a single partition).

• Queries that scan all columns of a table do not populate the cache (unless the table has only a single column).

• Queries on tables other than Hive, Paimon, Delta Lake, Hudi, or Iceberg do not populate the cache.

Use EXPLAIN VERBOSE to check whether a query populates the cache:

EXPLAIN VERBOSE SELECT col1 FROM hudi_table;

If the query plan shows dataCacheOptions={populate: false}, the query will not populate the cache. To force population:

SET populate_datacache_mode = 'always';

Footer Cache

StarRocks can cache Parquet Footer objects. Enable this feature with the following variable:

SET GLOBAL enable_file_metacache = true;

I/O adaptiveness

When I/O load on the cache disk is high, Data Cache automatically routes some requests to remote storage, using both local and remote I/O to increase throughput and reduce long-tail latency. This feature is enabled by default. To enable it manually:

SET GLOBAL enable_datacache_io_adaptor = true;

Monitor Data Cache

Query information_schema.be_datacache_metrics to view cache capacity and usage per BE node:

SELECT * FROM information_schema.be_datacache_metrics;

Monitor SQL cache hits

Monitor these query profile metrics to evaluate Data Cache performance: