Dynamic Table Incremental Refresh FAQ

Refresh failure due to excessive incremental data

The volume of incremental data processed per refresh can fluctuate. A sudden spike may cause the refresh to fail. Common reasons include:

• Overwriting one or more source tables can generate a large volume of data changes. To handle this, set the following GUC at the database level so that overwriting a source table automatically triggers an overwrite of the associated Dynamic Table.

ALTER DATABASE xxx SET hg_experimental_enable_dynamic_table_consume_change_when_base_overwrite = off;

• The Dynamic Table refresh was paused or inactive for a period, causing incremental data to accumulate.

• A business workload causes a short-term surge in data ingestion or changes on the source table.

Common error messages for this scenario include:

• Query OOM: ERRCODE_OUT_OF_MEMORY: Total memory used by all existing queries exceeded memory limitation.

• Worker OOM: SERVER_INTERNAL_ERROR message: "ERPC_ERROR_CONNECTION_CLOSED, reason: Ping timeout".

• BinaryArray Exceed 2G: internal error: Capacity error: BinaryArray cannot contain more than 2147483646 bytes.

To resolve this, run REFRESH OVERWRITE to perform a manual overwrite refresh. After the command completes, verify that subsequent incremental refreshes run normally.

REFRESH OVERWRITE TABLE dynamic_table_xxx WITH (refresh_mode = 'incremental', refresh_guc_hg_computing_resource = 'serverless');

Error: Multiple partitions / No partition is selected

Example error messages: Multiple partitions of table XXX is selected or No partition of table XXX is selected.

Cause: The source table is a physical partition table, but the Dynamic Table definition lacks a partition filter, so the query selects multiple physical sub-partitions.

Solution:

• Change the source table to use a logical partition. The Dynamic Table can then automatically select the correct partition.

• If the source table is a physical partition table and the Dynamic Table is also physically partitioned, the Dynamic Table definition must include a partition filter that ensures a one-to-one mapping between partitions.

Incremental refresh error: Cannot find index full id: XXX

Cause: Renaming a source table for an incremental Dynamic Table does not break incremental refresh. However, if the renamed table is later dropped, the refresh fails because the system can no longer locate the table by its internal ID.

Solution: Verify that all source tables listed in the error message exist, then run REFRESH OVERWRITE.

REFRESH OVERWRITE TABLE dynamic_table_xxx WITH (refresh_mode = 'incremental');

Incremental refresh error: GetLatestStreamOffset snapshots is empty or stream XXX is not existed

Cause: If a Dynamic Table is not refreshed for an extended period, the backend automatically clears the stored source table offset.

Solution: Run REFRESH OVERWRITE once.

REFRESH OVERWRITE TABLE dynamic_table_xxx WITH (refresh_mode = 'incremental');