Resource overview

更新时间:
复制 MD 格式

The Resource Overview tab in Data Lake Formation (DLF) gives you a unified view of storage consumption, compute resource usage, and request traffic for a data catalog or a specific Paimon table. Use it to track billing-related metrics, check whether QPS or bandwidth is approaching quota limits, and investigate traffic or latency anomalies.

View the resource overview

Data catalog level

  1. Log on to the DLF console.

  2. On the Data Catalog list page, click the name of the target catalog.

  3. Click the Resource Overview tab. The tab displays three metric groups: resource usage (billing-related), quota monitoring, and resource request details.

Table level

  1. On the Data Catalog list page, click the name of the target catalog, then click the target Paimon table in the destination database.

  2. Click the Resource Overview tab to view metrics and details for the current Paimon table.

Metric description (data catalog)

Resource usage

Use this section when you need to assess your current bill or plan for storage expansion.

MetricDescriptionData source
Total storage usageThe latest stored value.Direct measurement
Total storage usage (change)The change in total storage usage over the selected time period.Derived
Compute resource usage in the last 24 hours (CU × Hours)The total CU × Hours consumed by lake table optimization in the data catalog during the 24 hours before the selected end time.Direct measurement
Storage usage detailsThe raw and billable usage for each storage class over the selected time period. Storage classes: Standard, IA (IA storage class), Archive (Archive Storage), and Cold Archive.Direct measurement
Lake table optimization CU × Hours usageThe total CU × Hours consumed per hour by lake table optimization in the data catalog over the selected time period.Direct measurement
Number of file requestsThe number of file requests per hour over the selected time period. Write requests include billable PUT, POST, and HEAD requests. Read requests include billable GET requests.Direct measurement

Quota monitoring

Use this section when you want to confirm whether QPS or bandwidth is approaching its quota limit.

MetricDescriptionAggregationTime window
QPSThe average number of file requests per second.Average1 minute before the selected end time
Download bandwidthThe average download bandwidth.Average1 minute before the selected end time
Upload bandwidthThe average upload bandwidth.Average1 minute before the selected end time
QPS trendThe trend of the average number of file requests per second.Average per minuteLast minute
Bandwidth trendThe change trend of average upload and download bandwidth.Average per minuteLast minute

Resource request details

Use this section when you need to investigate abnormal request latency, traffic spikes, or identify which tables are consuming the most resources.

Traffic breakdown

MetricDescriptionData source
CU × Hours usage trend by categoryCU × Hours consumed by each task type over the selected time range.Direct measurement
CU × Hours usage by categoryTotal CU × Hours consumed by each task type over the selected time range.Direct measurement
Catalog read trafficTotal traffic, broken down by origin: Total (all traffic), Compaction (data compaction tasks), RequestLogs (system request logs), and DLF (internal system management such as lifecycle management and file cleanup).Direct measurement
Catalog write trafficTotal write traffic, broken down by origin (same categories as catalog read traffic).Direct measurement
Catalog read traffic by category (sampled by file type)Read traffic by file type: Manifest (manifest file), Snapshot (snapshot file), and Schema (metadata file).Sampling and aggregation
Catalog write traffic by category (sampled by file type)Write traffic by file type (same categories as catalog read traffic by category).Sampling and aggregation

Request counts

MetricDescriptionTime window
Catalog read requestsRead request count, broken down by origin: Compaction, RequestLogs, and DLF.Selected time range
Catalog write requestsWrite request count, broken down by origin (same categories as catalog read requests).Selected time range
Catalog read requests by category (sampled by file type)Read request count by file type: Manifest, Snapshot, Schema, and Bucket- (categorized by data format).Selected time range
Catalog write requests by category (sampled by file type)Write request count by file type (same categories as catalog read requests by category).Selected time range
Number of IO requestsThe total number of file requests.Per minute
Number of GetObject requestsThe number of GetObject requests.Per minute
Number of Upload requestsThe number of Upload requests.Per minute
Number of Delete requestsThe number of Delete requests.Per minute
Number of ObjectMeta requestsThe number of ObjectMeta requests.Per minute
Number of ListObjects requestsThe number of ListObjects requests.Per minute
Number of metadata requestsThe number of metadata requests.Per minute
Number of TableRead requestsThe number of TableRead requests.Per minute
Number of TableWrite requestsThe number of TableWrite requests.Per minute
Number of List requestsThe number of List requests.Per minute
Number of other metadata requestsThe number of other metadata requests.Per minute

Request latency

MetricDescriptionAggregationTime window
GetObject request latencyThe average latency of GetObject requests.AveragePer minute
Upload request latencyThe average latency of Upload requests.AveragePer minute
Delete request latencyThe average latency of Delete requests.AveragePer minute
ObjectMeta request latencyThe average latency of ObjectMeta requests.AveragePer minute
ListObjects request latencyThe average latency of ListObjects requests.AveragePer minute
TableRead request latencyThe average latency of TableRead requests.AveragePer minute
TableWrite request latencyThe average latency of TableWrite requests.AveragePer minute
List request latencyThe average latency of List requests.AveragePer minute
Other metadata request latencyThe average latency of other metadata requests.AveragePer minute

Top tables

Note

Top table data is generated through sampling and aggregation. Values are for reference only. For operations on multiple tables or at the database level, the table ID is displayed as default_id.

MetricDescriptionData source
Top tables by CU × HoursTables with the highest CU × Hours consumption over the selected time range.Sampling and aggregation
Top tables by read trafficTables with the highest read traffic over the selected time range.Sampling and aggregation
Top tables by write trafficTables with the highest write traffic over the selected time range.Sampling and aggregation
Top tables by read requestsTables with the highest number of read requests over the selected time range.Sampling and aggregation
Top tables by write requestsTables with the highest number of write requests over the selected time range.Sampling and aggregation

Metric description (table)

Note

Data collection and processing are subject to a delay. Values from the last three hours may fluctuate and are for reference only. Percentage data is generated by sampling and aggregation.

CU usage

Metrics in this group only include tables that use independent compaction.

MetricDescriptionTime granularity
Table CU × Hours (hourly)The hourly trend of CU × Hours consumed by the current table.Per hour
Table CU × Hours (daily)The daily trend of CU × Hours consumed by the current table.Per day
Current table CU × Hours trend by categoryCU × Hours consumed by each task type for the current table.Over the selected time range
Current table CU × Hours by categoryTotal CU × Hours consumed by each task type for the current table.Over the selected time range
Current table/Catalog total CU usage percentage trendThe percentage of the data catalog's total CU usage consumed by the current table, shown as a trend.Over the selected time range
Current table/Catalog total CU usage percentageThe percentage of the data catalog's total CU usage consumed by the current table.Over the selected time range

Traffic breakdown

MetricDescriptionData source
Table read trafficThe percentage of total table read traffic attributed to each origin request.Direct measurement
Table write trafficThe percentage of total table write traffic attributed to each origin request.Direct measurement
Table read traffic by category (sampled by origin request)The percentage of table read traffic from each origin request category.Sampling and aggregation
Table write traffic by category (sampled by origin request)The percentage of table write traffic from each origin request category.Sampling and aggregation
Table/Catalog read request percentageThe percentage of the data catalog's total requests that come from the current table (reads).Direct measurement
Table/Catalog write request percentageThe percentage of the data catalog's total requests that come from the current table (writes).Direct measurement

Request counts

MetricDescriptionData source
Table read requestsThe percentage of total table read requests attributed to each origin request.Direct measurement
Table write requestsThe percentage of total table write requests attributed to each origin request.Direct measurement
Table read requests by category (sampled by origin request)The percentage of table read requests from each origin request category.Sampling and aggregation
Table write requests by category (sampled by origin request)The percentage of table write requests from each origin request category.Sampling and aggregation
Table read requests by category (sampled by file type)The percentage of table read requests for each file type.Sampling and aggregation
Table write requests by category (sampled by file type)The percentage of table write requests for each file type.Sampling and aggregation
Number of GetObject requestsThe total number of GetObject requests.Direct measurement
Number of Upload requestsThe total number of Upload requests.Direct measurement
Number of Delete requestsThe total number of Delete requests.Direct measurement
Number of ObjectMeta requestsThe total number of ObjectMeta requests.Direct measurement
Number of ListObjects requestsThe total number of ListObjects requests.Direct measurement

Request latency

MetricDescriptionAggregation
GetObject request latencyThe average latency of GetObject requests.Average
Upload request latencyThe average latency of Upload requests.Average
Delete request latencyThe average latency of Delete requests.Average
ObjectMeta request latencyThe average latency of ObjectMeta requests.Average
ListObjects request latencyThe average latency of ListObjects requests.Average