Liquid Table

What is Liquid Table

Liquid Table is a new table storage mode in Hologres. Its defining capability:

This addresses long-standing pain points in traditional data warehouses where adjusting shard granularity after data growth is prohibitively expensive:

Key capabilities

• Online Resharding: Use ALTER TABLE to change the shard count at any time, transparent to applications.

• Multiple storage formats: Supports columnar (column), row-oriented (row), and hybrid (row,column).

• Controlled data redistribution: Choose between automatic background redistribution or skipping redistribution for faster metadata switching.

• Partition window control: Limit redistribution to the most recent N days/months of partitions, reducing I/O overhead.

• Broad feature compatibility: Works with Dynamic Table, GSI, full-text index, vector index (HGraph), Time Travel (MVCC), Binlog, and logical partition tables.

Typical use cases

1. Data growth outpaces the initial shard estimate: Start with 16 shards, scale to 64 six months later.

2. Temporary scale-out for traffic events: Add shards before a sales event for higher query concurrency, then scale in afterward to reclaim resources.

3. Hot/cold partition management: Scale in cold historical partitions while keeping hot partitions scaled out.

4. AI vector search with elastic scaling: Dynamically scale vector tables as recall volume grows; indexes remain available throughout.

Create a Liquid Table

Add WITH (liquid_table = 'true') to your CREATE TABLE statement:

-- Columnar storage (default, for OLAP workloads)
CREATE TABLE my_table (
    id   INT NOT NULL,
    name TEXT,
    ts   TIMESTAMPTZ,
    PRIMARY KEY(id)
) WITH (liquid_table = 'true');

-- Row-oriented storage (for point lookups / high-frequency updates)
CREATE TABLE my_row_table (
    id   INT NOT NULL,
    name TEXT,
    PRIMARY KEY(id)
) WITH (liquid_table = 'true', orientation = 'row');

-- Hybrid storage (for mixed point-lookup + OLAP workloads)
CREATE TABLE my_hybrid_table (
    id   INT NOT NULL,
    name TEXT,
    PRIMARY KEY(id)
) WITH (liquid_table = 'true', orientation = 'row,column');

Change the shard count (Resharding)

Resharding is performed by reassigning the table to a different Table Group:

-- Create a target Table Group if one does not already exist
CALL hg_create_table_group('tg_64', 64);

-- Scale out (takes effect synchronously)
ALTER TABLE my_table SET (table_group = 'tg_64');

-- Scale in
ALTER TABLE my_table SET (table_group = 'tg_16');

Verify the current shard configuration

-- Check the Table Group and shard count for a table
SELECT property_key, property_value
FROM hologres.hg_table_properties
WHERE table_name = 'my_table'
AND property_key = 'table_group';

-- Check Table Group metadata
SELECT * FROM hologres.hg_table_group_properties
WHERE tablegroup_name = 'tg_64';

Creation-time properties

Choosing the right value for liquid_table_enable_data_reorganization

Partition window example

-- Redistribute only the most recent 30 days of partitions (requires a time-type partition key)
ALTER TABLE order_log SET (
    table_group = 'tg_64',
    liquid_table_reorganization_partition_window = '30 day'
);

This is ideal for long-retention log or fact tables where historical cold partitions are rarely accessed — only the hot partitions need redistribution.

Compatibility matrix (V4.2.0)

Index coexistence examples

-- Full-text index
CREATE INDEX ft_idx ON my_table USING FULLTEXT (content)
    WITH (tokenizer = 'jieba');

-- GSI (Global Secondary Index)
CREATE GLOBAL INDEX gsi_name ON my_table(name) INCLUDE (ts);

-- Vector index (via the vectors property)
ALTER TABLE my_table SET (vectors = '{
    "embedding": {
        "algorithm": "HGraph",
        "distance_method": "Cosine"
    }
}');

After creating indexes on a Liquid Table, you can perform Resharding via ALTER TABLE ... SET (table_group = '...') directly. All indexes remain usable on the new shards automatically.

Impact summary

Impact on writes

1) Writes are not expected to fail, but configure client retries. During the metadata switch, a small number of concurrent requests may encounter transient errors (e.g. connection timeouts). Connectors should have automatic retries enabled. There will be no prolonged write unavailability.

2) Write latency will spike observably — this is not "zero impact". In-flight writes must wait for the new topology to take effect. Spikes are typically in the millisecond-to-second range but may be longer when:

• Long-running transactions are active on the table.

• Instance I/O utilization is already high.

• The scaling factor is very large (e.g. 8 shards directly to 128 shards).

3) During the redistribution phase, writes are not blocked but latency may increase slightly.

• Background data migration consumes instance I/O and may compete with foreground writes for resources.

• On instances with high I/O utilization, write latency during redistribution may be slightly higher than steady state, and throughput may also decrease slightly.

• We recommend continuously monitoring instance I/O and write RT metrics during the redistribution phase.

4) For COPY / bulk import workloads: start bulk imports only after Resharding and redistribution are fully complete. Importing large volumes during Resharding significantly extends redistribution time and amplifies write latency fluctuations.

Impact on queries

1) Queries are not expected to fail or be blocked.

• During the metadata switch, new queries execute against the new topology.

• In-flight queries complete normally.

• In rare cases, queries issued at the exact switch instant may time out due to route rerouting. Client retries should handle this.

2) Read latency impact at the switch instant is typically minimal, but not zero.

• Compared to writes, the read path is less affected by the switch.

• Queries hitting the switch moment undergo route rerouting; latency-sensitive real-time APIs may observe millisecond-level jitter.

• For high-QPS point-query scenarios, plan upstream timeout settings assuming brief jitter will occur.

3) Old-data query performance may degrade significantly until redistribution completes:

4) Queries on newly written data are largely unaffected.

• New data is written to the new shard topology; query performance is consistent with steady state.

• However, if a query spans both new and old data (e.g. time-range scans), the overall RT is still dragged down by the degraded old-data performance.

5) Index queries (GSI / full-text / vector) remain functional but may not have stable performance. Indexes remain usable after Resharding, but physical I/O for index lookups may contend with background redistribution, causing elevated response times until redistribution completes. Monitor P99 and long-tail queries closely.

Recommendations for latency-sensitive workloads

If your workload falls into any of the following categories, treat Resharding as a risky change and follow your change management process:

Estimated impact duration (conservative)

Bottom line

Default behavior

Liquid Tables with Binlog enabled do not allow Resharding by default. Running ALTER TABLE ... SET (table_group = ...) directly returns an error.

Forced Resharding procedure

-- Step 1: Enable the experimental GUC (session-level only)
SET hg_experimental_enable_liquid_resharding_for_table_with_binlog = on;

-- Step 2: Perform Resharding
ALTER TABLE my_binlog_table SET (table_group = 'tg_64');

-- Step 3: Clean up delta data on old shards
CALL hg_liquid_resharding_drop_non_current_delta('my_binlog_table');

Impact on Binlog consumers

Approach 1: Convert via REBUILD (recommended)

Starting from Hologres V4.2, you can use ASYNC REBUILD TABLE to rebuild an existing table into a Liquid Table in place, with no new table, no manual data migration, and the table name unchanged. For details about REBUILD, see REBUILD.

-- Convert an existing table into a Liquid Table in place (table name unchanged)
ASYNC REBUILD TABLE my_table
SET (
    liquid_table = 'true'
);

Approach 2: New-table migration

Create a new Liquid Table, migrate the data from the existing table, and switch over using an atomic transactional rename.

-- 1. Create a new Liquid Table with the same schema
CREATE TABLE my_table_new (
    id   INT NOT NULL,
    name TEXT,
    ts   TIMESTAMPTZ,
    PRIMARY KEY(id)
) WITH (liquid_table = 'true');

-- 2. Copy data
INSERT INTO my_table_new SELECT * FROM my_table_old;

-- 3. Recreate indexes (if any)
CREATE INDEX ... ON my_table_new ...;

-- 4. Atomic swap using a transaction
BEGIN;
ALTER TABLE my_table_old RENAME TO my_table_bak;
ALTER TABLE my_table_new RENAME TO my_table;
COMMIT;

-- 5. Drop the backup after verification
DROP TABLE my_table_bak;

When to enable Liquid Table

Strongly recommended:

• Data volume is expected to grow 5x or more over the table's lifetime.

• Traffic has seasonal spikes or sales events requiring elastic shard scaling.

• Core fact or wide tables in a real-time data warehouse.

• AI vector search tables.

Can defer:

• Small dimension tables with stable data volume.

• Workloads that depend on physical partition tables.

• Workloads that depend on MaxCompute direct read.

Shard count planning guidelines

Resharding execution tips

1. Run during off-peak hours: Execute ALTER during a low-traffic maintenance window.

2. Quiesce large DML: Ensure no bulk INSERT/UPDATE/DELETE jobs are running.

3. Prefer power-of-2 multiples: e.g. 16 → 32 → 64 yields smoother performance than 16 → 50.

4. Use partition window for large tables: Set liquid_table_reorganization_partition_window to limit the redistribution scope on long-lifecycle partition tables.

5. Rehearse Binlog tables end-to-end: Test the complete Resharding + downstream restart flow in a staging environment first.

Monitoring recommendations

• Before redistribution completes: periodically compare query response times across old and new shards to gauge progress.

• Instance resource utilization: Resharding incurs additional I/O overhead — monitor CPU and I/O usage.

• Binlog cursor lag: for Binlog-enabled tables, monitor downstream consumer lag.