Beam storage engine

Beam is a next-generation in-house storage engine developed based on the table access methods of PostgreSQL 12 for AnalyticDB for PostgreSQL.

The Beam storage engine consists of two parts:

• A row-oriented Delta storage that handles real-time writes.

• A PAX-based column-oriented Base storage that handles batch writes and large-scale scanning.

Compared with row-oriented heap tables, Beam tables require fewer disk I/O operations and deliver significantly better query performance in analytical scenarios. The Beam storage engine supports primary keys, write deduplication, and concurrent update and delete operations. You can use Data Transmission Service (DTS) to synchronize data to Beam tables. With a single copy of storage, Beam handles both OLTP and OLAP workloads, eliminating the need to synchronize data between row-oriented and column-oriented engines.

Usage notes

• Only AnalyticDB for PostgreSQL V7.0.x instances in elastic storage mode support the Beam storage engine.

• The Beam storage engine is officially available starting from AnalyticDB for PostgreSQL V7.0.6.2 in elastic storage mode. This version fixes issues identified during the public preview. We recommend that you update your instance to V7.0.6.2 or later.

Features

High-performance real-time writes

The Beam storage engine consists of a row-oriented Delta storage and a PAX-based column-oriented Base storage. When data is written, the appropriate storage mode is selected based on the write method. For real-time streaming writes such as INSERT INTO VALUES, data is written to the row-oriented Delta storage, delivering write performance comparable to row-oriented heap tables.

High-throughput batch import

For batch processing methods such as COPY or INSERT INTO SELECT, data is written to the column-oriented Base storage for higher throughput.

High-performance OLAP query

The Beam storage engine uses the following methods to optimize query performance:

• Column pruning

• Multiple compression algorithms

• Zone map filtering

• I/O prefetch

These methods reduce disk I/O, increase I/O utilization, and improve query performance.

Primary key and write deduplication

The Beam storage engine supports PostgreSQL primary keys, allowing you to build primary key indexes for Beam tables for data deduplication, similar to heap tables. Beam also supports the UPSERT syntax.

DTS synchronization

With primary keys and DTS, you can synchronize data to Beam tables in the same way as heap tables. Set the destination table to a Beam table in a DTS task to achieve better query performance without synchronizing data between separate row-oriented and column-oriented tables.

References

• For information about how to use the Beam storage engine, see Beam usage.

• The Beam storage engine supports multiple compression algorithms. The dictionary encoding feature can compress string data to integer data to improve storage efficiency and accelerate filter-based and aggregate-based queries. For more information, see Dictionary encoding.

• If you frequently perform range queries or equivalence filtering on one or more columns, you can specify a Beam sort key to improve query performance. For more information, see Beam sorting optimization (V7.0).