Manage refresh tasks

更新时间:
复制 MD 格式

Dynamic tables run refresh tasks in the background based on the configured start time and refresh interval. When the base table changes, the dynamic table refreshes to reflect the latest data. This topic explains how to monitor, view, and manage those refresh tasks.

Monitoring metrics

Starting from Hologres V4.0.8, the following four metrics are available for dynamic table refresh tasks. Configure alert thresholds in Cloud Monitor based on your requirements. For more information, see CloudMonitor.

Instance-level refresh failure QPS

Metric name: Dynamic Table Refresh Failed QPS Unit: count/s (queries per second)

This metric shows the failure rate of refresh tasks across all dynamic tables in an instance. Under normal conditions, the value stays close to zero.

If this value is consistently nonzero or rising, go to HoloWeb for troubleshooting.

Data latency

Metric name: Dynamic Table Lag Unit: s (seconds)

This metric shows how far behind each dynamic table is relative to its base table or a specified point in time. It reflects data freshness.

If latency keeps increasing:

Possible cause

Next step

Refresh tasks are failing repeatedly, or auto-refresh is paused

Go to HoloWeb console and check the task status

A large volume of upstream data has changed and instance resources are insufficient to keep up

Review Hologres monitoring metrics such as refresh duration to investigate resource bottlenecks

Refresh task duration

Metric name: Dynamic Table Refresh Duration Unit: ms (milliseconds)

This metric shows how long the current refresh task for each dynamic table has been running. Use it to detect whether the refresh epoch is lengthening.

If the value suddenly increases or stays significantly above its historical average:

Possible cause

Next step

Resource bottleneck

Check instance CPU, memory, and storage metrics

Increase in upstream data volume

Query hologres.hg_dynamic_table_refresh_history to compare the current duration against historical baselines

Per-table refresh failure QPM

Metric name: Dynamic Table Refresh Failed QPM Unit: count/m (count per minute)

This metric shows the number of failed refresh tasks per minute for each dynamic table. Under normal conditions, the value is zero.

Interpreting this metric:

Observed pattern

Meaning

Action

Occasional spikes, subsequent refreshes succeed

Transient system pressure or instance upgrade

No action needed

Value stays above zero persistently for a specific table

Persistent refresh failure

Check the table's failure log for the error message and resolve the underlying issue

View refresh tasks

View running refresh tasks

Using hologres.hg_dynamic_table_refresh_activity

The hologres.hg_dynamic_table_refresh_activity system table shows running refresh tasks — including full and incremental refreshes — along with their resource consumption. For field descriptions, see hologres.hg_dynamic_table_refresh_activity.

This system table is supported only in Hologres V3.0, and V4.0.8 and later.
-- View currently running refresh tasks
SELECT
    pid,
    query_id,
    refresh_mode,
    'RUNNING' AS status,
    refresh_start,
    extract(epoch FROM duration) AS duration,                            -- milliseconds
    serverless_queue_time_ms::bigint / 1000 AS serverless_queue_time_sec,
    serverless_resource_used_time_ms::bigint / 1000 AS serverless_resource_used_time_sec,
    serverless_allocated_cores
FROM hologres.hg_dynamic_table_refresh_activity
WHERE datname = '${database}'
  AND table_write = quote_ident('${schema}') || '.' || quote_ident('${tableName}')
ORDER BY refresh_start DESC
LIMIT 2000;

Using hg_stat_activity

The hg_stat_activity system view also shows running refresh tasks. The display differs by refresh mode:

  • Full refresh: An INSERT statement appears.

  • Incremental refresh: A Refresh task appears.

Using monitoring metrics

Check metrics such as QPS, RPS (records per second), and latency to confirm execution status. A Command Type of refresh indicates a dynamic table refresh task. For more information, see Metrics.

If the refresh task runs on Serverless Computing resources, you can also check its status in the Serverless Computing metrics.

View historical refresh tasks

Using hologres.hg_dynamic_table_refresh_history

The hologres.hg_dynamic_table_refresh_history system table records the history of all refresh tasks — full, incremental, and manual — for the past month. For field descriptions, see hologres.hg_dynamic_table_refresh_history.

Records are retained for one month. Data older than one month cannot be queried.
Table owners can view only their own refresh history. Users with the superuser role can view all refresh records.

Example 1: View incremental refresh tasks from the past day

SELECT
    query_id,
    refresh_mode,
    status,
    refresh_start,
    duration,
    refresh_latency / 1000 AS refresh_latency_second,
    serverless_allocated_cores,
    queue_time_ms::bigint / 1000 AS queue_time_second,
    serverless_resource_used_time_ms::bigint / 1000 AS serverless_resource_used_time_second
FROM hologres.hg_dynamic_table_refresh_history
WHERE refresh_start >= CURRENT_DATE - INTERVAL '1 day'
  AND dynamic_table_name = '<dynamic_table>'
  AND refresh_mode = 'incremental'
ORDER BY refresh_start DESC
LIMIT 100;

Example 2: View all refresh tasks in the instance from the past day

SELECT
    query_id,
    refresh_mode,
    status,
    refresh_start,
    duration,
    refresh_latency / 1000 AS refresh_latency_second,
    serverless_allocated_cores,
    queue_time_ms::bigint / 1000 AS queue_time_second,
    serverless_resource_used_time_ms::bigint / 1000 AS serverless_resource_used_time_second
FROM hologres.hg_dynamic_table_refresh_history
WHERE refresh_start >= CURRENT_DATE - INTERVAL '1 day';

Example 3: View refresh tasks for a specific table from the past day

SELECT
    query_id,
    refresh_mode,
    status,
    refresh_start,
    duration,
    refresh_latency / 1000 AS refresh_latency_second,
    serverless_allocated_cores,
    queue_time_ms::bigint / 1000 AS queue_time_second,
    serverless_resource_used_time_ms::bigint / 1000 AS serverless_resource_used_time_second
FROM hologres.hg_dynamic_table_refresh_history
WHERE schema_name = '<schema_name>'
  AND dynamic_table_name = '<dynamic_table>'
  AND refresh_start >= CURRENT_DATE - INTERVAL '1 day';
For full-refresh dynamic tables created with the legacy 3.0 syntax, hologres.hg_dynamic_table_refresh_history may not accurately reflect the success or failure status — a failed refresh may appear as Success. To retrieve the true refresh history for these tables: 1. Get the cron_job_name from hologres.hg_dynamic_table_properties. 2. Query cron job execution records using that name.
-- Step 1: Get the cron_job_name
SELECT property_value AS cron_job_name
FROM hologres.hg_dynamic_table_properties
WHERE dynamic_table_name = '<dt_name>'
  AND property_key = 'cron_job_name';

-- Step 2: Query cron job execution records
SELECT *
FROM hologres.hg_user_cron_tasks
WHERE jobname = '<cron_job_name>'
ORDER BY start_time DESC;

Using slow query logs

Refresh tasks also appear in slow query logs with Command Type set to refresh. For more information, see Get and analyze slow query logs.

View the execution plan of a refresh task

Use EXPLAIN and EXPLAIN ANALYZE on a refresh statement to view its execution plan and identify performance bottlenecks, the same way you would for a regular query.

EXPLAIN REFRESH DYNAMIC TABLE hmtest.dt_order_lineitem;

Example output:

                                                                                              QUERY PLAN
-------------------------------------------------------------------------------------------------------------
 Gather  (cost=0.00..10.13 rows=1 width=16)
   ->  Insert  (cost=0.00..10.13 rows=1 width=16)
         ->  Redistribution  (cost=0.00..10.11 rows=1 width=16)
               ->  Final HashAggregate  (cost=0.00..10.11 rows=1 width=16)
                     Group Key: orders.o_orderpriority
                     ->  Redistribution  (cost=0.00..10.11 rows=10 width=16)
                           Hash Key: orders.o_orderpriority
                           ->  Partial HashAggregate  (cost=0.00..10.11 rows=10 width=16)
                                 Group Key: orders.o_orderpriority
                                 ->  Hash Left Semi Join  (cost=0.00..10.11 rows=1000 width=8)
                                       Hash Cond: (orders.o_orderkey = lineitem.l_orderkey)
                                       ->  Redistribution  (cost=0.00..5.03 rows=1000 width=16)
                                             Hash Key: orders.o_orderkey
                                             ->  Local Gather  (cost=0.00..5.01 rows=1000 width=16)
                                                   ->  Seq Scan on orders  (cost=0.00..5.01 rows=1000 width=16)
                                                         Filter: ((o_orderdate >= '1996-07-01 00:00:00+08'::timestamp with time zone) AND (o_orderdate < '1996-10-01 00:00:00+08'::timestamp with time zone))
                                       ->  Hash  (cost=5.03..5.03 rows=1000 width=8)
                                             ->  Redistribution  (cost=0.00..5.03 rows=1000 width=8)
                                                   Hash Key: lineitem.l_orderkey
                                                   ->  Local Gather  (cost=0.00..5.03 rows=1000 width=8)
                                                         ->  Seq Scan on lineitem  (cost=0.00..5.03 rows=1000 width=8)
                                                               Filter: (l_commitdate < l_receiptdate)
 Optimizer: HQO version 2.1.0
(23 rows)

Set the refresh timeout duration

Set a timeout to prevent long-running refresh tasks from blocking resources. Hologres supports three levels of timeout configuration.

Table-level timeout

Set a table-level timeout when creating the dynamic table. It applies to all refresh tasks for that table. The following example uses the tpch_10g public dataset. Before running the code, import the dataset. For more information, see Create a public dataset import task.

-- Set a 30-minute timeout for all refresh tasks on this table
CREATE DYNAMIC TABLE tpch_q1_batch
WITH (
    refresh_mode = 'full',
    auto_refresh_enable = 'true',
    full_auto_refresh_interval = '1 hours',
    refresh_guc_statement_timeout = '30 mins'
)
AS
SELECT
    l_returnflag,
    l_linestatus,
    SUM(l_quantity) AS sum_qty,
    SUM(l_extendedprice) AS sum_base_price,
    SUM(l_extendedprice * (1 - l_discount)) AS sum_disc_price,
    SUM(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge,
    AVG(l_quantity) AS avg_qty,
    AVG(l_extendedprice) AS avg_price,
    AVG(l_discount) AS avg_disc,
    COUNT(*) AS count_order
FROM hologres_dataset_tpch_10.lineitem
WHERE l_shipdate <= DATE '1998-12-01' - INTERVAL '120' DAY
GROUP BY l_returnflag, l_linestatus;

Session-level timeout

For a manual refresh, set the timeout using a session-level GUC (Grand Unified Configuration) parameter:

SET statement_timeout = <time>;
REFRESH DYNAMIC TABLE <dynamic_schema_name.dynamic_table_name>;

For more information about timeout settings, see Modify active query timeout.

Per-refresh timeout

Override the timeout for a single manual refresh using refresh ... WITH (refresh_guc_statement_timeout = '...'):

REFRESH DYNAMIC TABLE <schema_name.table_name> WITH (
    refresh_guc_statement_timeout = '30 mins'
);

Trigger a manual refresh

Run the following statement to trigger an immediate refresh:

REFRESH DYNAMIC TABLE <schema_name.table_name>;
If auto-refresh is enabled, a manual refresh runs in parallel with the scheduled auto-refresh task. Both complete normally. The system retains only one copy of the latest data.

Cancel a refresh task

Dynamic tables created with the new 3.1 syntax

Cancel a running refresh task

Query the query_job_id of the running task from hologres.hg_dynamic_table_refresh_log, then cancel it using hologres.hg_internal_cancel_query_job.

-- Step 1: Get the query_job_id
SELECT query_job_id
FROM hologres.hg_dynamic_table_refresh_log('<dt_name>')
WHERE status = 'Running';

-- Step 2: Cancel the task
SELECT hologres.hg_internal_cancel_query_job('<query_job_id>');
Only a superuser can cancel a refresh task using hologres.hg_internal_cancel_query_job.

Pause auto-refresh for a table

To stop all subsequent refresh tasks at the table level, disable auto-refresh:

ALTER DYNAMIC TABLE [IF EXISTS] [<schema>.]<table_name> SET (auto_refresh_enable = false);
Important

This operation stops all future refresh tasks for the table. Data in the dynamic table will not be updated until auto-refresh is re-enabled. To re-enable it, see ALTER DYNAMIC TABLE.

Dynamic tables created with the 3.0 syntax

Cancel a running refresh task

If a refresh task is taking too long or appears stuck, cancel it using pg_cancel_backend.

-- Cancel the refresh task by its process ID (pid)
SELECT pg_cancel_backend(<pid>);

Get the pid from hologres.hg_dynamic_table_refresh_activity or hg_stat_activity. For more information, see View refresh tasks

To cancel refresh tasks in batches, use the same method as for regular queries. For more information, see Terminate a query.

Pause auto-refresh for a table

To stop all subsequent refresh tasks at the table level:

ALTER DYNAMIC TABLE [IF EXISTS] [<schema>.]<table_name> SET (auto_refresh_enable = false);
Important

This operation stops all future refresh tasks for the table. Data in the dynamic table will not be updated until auto-refresh is re-enabled. To re-enable it, see ALTER DYNAMIC TABLE.