compute_usage system table
The compute_usage system table records per-table CU-Hours consumed by table optimization, aggregated by hour. Query it with Flink or another supported engine to analyze costs and optimize resource usage. This system table applies only to Paimon Catalogs.
-
By default,
compute_usageretains data from the most recent 30 days. You can change partition retention in the Storage Optimization section. For details, see Partition lifecycle management. Partitions outside the configured retention period expire and are automatically cleaned up. To keep data long term, archive it periodically to a table you manage. -
The CU-Hours in this table are intended for consumption analysis and trend monitoring. Refer to your bill for the final charges.
Field reference
The following is the built-in schema definition. It is shown for reference only and does not need to be executed manually.
CREATE TABLE `compute_usage` (
`database_name` STRING COMMENT 'Database name',
`table_id` STRING COMMENT 'Table ID',
`table_name` STRING COMMENT 'Table name',
`cu` DOUBLE COMMENT 'CU * Hours consumed',
`start_time` BIGINT COMMENT 'Usage period start time (epoch milliseconds)',
`end_time` BIGINT COMMENT 'Usage period end time (epoch milliseconds)',
`dt` STRING COMMENT 'Date (yyyyMMdd)'
) COMMENT 'Hourly table-level compute resource consumption'
PARTITIONED BY (dt) WITH (
'file.format' = 'avro',
'partition.expiration-time' = '30 d',
'partition.timestamp-formatter' = 'yyyyMMdd'
);
The fields are defined as follows:
|
Field |
Description |
|
database_name |
The database name. |
|
table_id |
The table ID. For multi-table operations, database-level operations, and similar scenarios, the value is |
|
table_name |
The table name. For multi-table operations, database-level operations, and similar scenarios, the value is |
|
cu |
The CU-Hours consumed during the hour. This is the total across all lakehouse table optimization jobs, including intelligent compaction and lifecycle management. |
|
start_time |
The start time of the metering period, as an epoch millisecond timestamp. Typically aligned to the top of the hour. |
|
end_time |
The end time of the metering period, as an epoch millisecond timestamp. Typically aligned to the next top of the hour. |
|
dt |
The partition field. The date format is |
Query examples
You can run SQL queries using DLF data preview and external compute engines such as Realtime Compute for Apache Flink, EMR Serverless Spark, and EMR Serverless StarRocks:
DLF data preview
You can run SQL queries directly using the DLF data preview and discovery features without setting up a separate compute engine. For details and pricing, see Data preview.
Flink SQL
To run the SQL examples using Flink, a DLF catalog must be configured in Flink. For detailed configuration steps, see Access DLF with Flink SQL.
Spark
To run SQL queries using EMR Serverless Spark, see Access DLF with Serverless Spark.
StarRocks
Bind the EMR Serverless StarRocks instance to a DLF catalog. In StarRocks Manager, associate the user who runs queries with a RAM user that has DLF permissions. For more information, see Access DLF with Serverless StarRocks.
Rank per-table compute consumption for a given day
SELECT
database_name,
table_name,
table_id,
SUM(cu) AS total_cu
FROM `<yourCatalogName>`.`system`.`compute_usage`
WHERE dt = '<date>' -- Replace with the actual query date in yyyyMMdd format
GROUP BY database_name, table_name, table_id
ORDER BY total_cu DESC;
Track total compute consumption for a catalog
By day
SELECT
dt,
SUM(cu) AS daily_cu,
min(start_time) AS min_start_time,
max(start_time) AS max_start_time
FROM `<yourCatalogName>`.`system`.`compute_usage`
WHERE dt >= '<startDate>' -- Replace with the actual start date
AND dt <= '<endDate>' -- Replace with the actual end date
GROUP BY dt
ORDER BY dt;
By hour
SELECT
start_time,
end_time,
SUM(cu) AS hourly_cu
FROM `<yourCatalogName>`.`system`.`compute_usage`
WHERE dt = '<date>' -- Replace with the actual query date
GROUP BY start_time, end_time
ORDER BY start_time;
Track compute consumption for a single table
By day
SELECT
dt,
SUM(cu) AS daily_cu,
min(start_time) AS min_start_time,
max(start_time) AS max_start_time
FROM `<yourCatalogName>`.`system`.`compute_usage`
WHERE table_id = '<yourTableId>'
AND dt >= '<startDate>' -- Replace with the actual start date
AND dt <= '<endDate>' -- Replace with the actual end date
GROUP BY dt
ORDER BY dt;
By hour
SELECT
start_time,
end_time,
SUM(cu) AS hourly_cu
FROM `<yourCatalogName>`.`system`.`compute_usage`
WHERE table_id = '<yourTableId>'
AND dt = '<date>' -- Replace with the actual query date
GROUP BY start_time, end_time
ORDER BY start_time;