compute_usage system table

Updated at:

The compute_usage system table records per-table CU-Hours consumed by table optimization, aggregated by hour. Query it with Flink or another supported engine to analyze costs and optimize resource usage. This system table applies only to Paimon Catalogs.

Note
  • By default, compute_usage retains data from the most recent 30 days. You can change partition retention in the Storage Optimization section. For details, see Partition lifecycle management. Partitions outside the configured retention period expire and are automatically cleaned up. To keep data long term, archive it periodically to a table you manage.

  • The CU-Hours in this table are intended for consumption analysis and trend monitoring. Refer to your bill for the final charges.

Field reference

Note

The following is the built-in schema definition. It is shown for reference only and does not need to be executed manually.

CREATE TABLE `compute_usage` (
  `database_name` STRING  COMMENT 'Database name',
  `table_id`      STRING  COMMENT 'Table ID',
  `table_name`    STRING  COMMENT 'Table name',
  `cu`            DOUBLE  COMMENT 'CU * Hours consumed',
  `start_time`    BIGINT  COMMENT 'Usage period start time (epoch milliseconds)',
  `end_time`      BIGINT  COMMENT 'Usage period end time (epoch milliseconds)',
  `dt`            STRING  COMMENT 'Date (yyyyMMdd)'
) COMMENT 'Hourly table-level compute resource consumption'
PARTITIONED BY (dt) WITH (
   'file.format' = 'avro',
   'partition.expiration-time' = '30 d',
   'partition.timestamp-formatter' = 'yyyyMMdd'
);

The fields are defined as follows:

Field

Description

database_name

The database name.

table_id

The table ID. For multi-table operations, database-level operations, and similar scenarios, the value is multi-table-job-default-table-id.

table_name

The table name. For multi-table operations, database-level operations, and similar scenarios, the value is multi-table-job-default-table-name.

cu

The CU-Hours consumed during the hour. This is the total across all lakehouse table optimization jobs, including intelligent compaction and lifecycle management.

start_time

The start time of the metering period, as an epoch millisecond timestamp. Typically aligned to the top of the hour.

end_time

The end time of the metering period, as an epoch millisecond timestamp. Typically aligned to the next top of the hour.

dt

The partition field. The date format is yyyyMMdd. By default, data from the most recent 30 days is retained.

Query examples

You can run SQL queries using DLF data preview and external compute engines such as Realtime Compute for Apache Flink, EMR Serverless Spark, and EMR Serverless StarRocks:

DLF data preview

You can run SQL queries directly using the DLF data preview and discovery features without setting up a separate compute engine. For details and pricing, see Data preview.

Flink SQL

To run the SQL examples using Flink, a DLF catalog must be configured in Flink. For detailed configuration steps, see Access DLF with Flink SQL.

Spark

To run SQL queries using EMR Serverless Spark, see Access DLF with Serverless Spark.

StarRocks

Bind the EMR Serverless StarRocks instance to a DLF catalog. In StarRocks Manager, associate the user who runs queries with a RAM user that has DLF permissions. For more information, see Access DLF with Serverless StarRocks.

Rank per-table compute consumption for a given day

SELECT
    database_name,
    table_name,
    table_id,
    SUM(cu) AS total_cu
FROM `<yourCatalogName>`.`system`.`compute_usage`
WHERE dt = '<date>'  -- Replace with the actual query date in yyyyMMdd format
GROUP BY database_name, table_name, table_id
ORDER BY total_cu DESC;

Track total compute consumption for a catalog

By day

SELECT
    dt,
    SUM(cu) AS daily_cu,
    min(start_time) AS min_start_time,
    max(start_time) AS max_start_time
FROM `<yourCatalogName>`.`system`.`compute_usage`
WHERE dt >= '<startDate>'  -- Replace with the actual start date
    AND dt <= '<endDate>'  -- Replace with the actual end date
GROUP BY dt
ORDER BY dt;

By hour

SELECT
    start_time,
    end_time,
    SUM(cu) AS hourly_cu
FROM `<yourCatalogName>`.`system`.`compute_usage`
WHERE dt = '<date>'  -- Replace with the actual query date
GROUP BY start_time, end_time
ORDER BY start_time;

Track compute consumption for a single table

By day

SELECT
    dt,
    SUM(cu) AS daily_cu,
    min(start_time) AS min_start_time,
    max(start_time) AS max_start_time
FROM `<yourCatalogName>`.`system`.`compute_usage`
WHERE table_id = '<yourTableId>'
    AND dt >= '<startDate>'  -- Replace with the actual start date
    AND dt <= '<endDate>'  -- Replace with the actual end date
GROUP BY dt
ORDER BY dt;

By hour

SELECT
    start_time,
    end_time,
    SUM(cu) AS hourly_cu
FROM `<yourCatalogName>`.`system`.`compute_usage`
WHERE table_id = '<yourTableId>'
    AND dt = '<date>'  -- Replace with the actual query date
GROUP BY start_time, end_time
ORDER BY start_time;