Resource observation

Procedure

1. Log on to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, click Resource Observation.

3. On the Resource Observation page, click the Computing Resource tab.

4. Select a level-1 quota, a time range, and a time interval.

   The time interval specifies the data point frequency. You can select an adaptive interval, or a 1-minute, 5-minute, or 15-minute interval. To ensure performance, the system automatically sets the interval to adaptive if the time range exceeds 72 hours.

5. Click the icon to the left of a target level-2 quota to view its resource consumption trend chart. You can expand multiple charts at the same time.

6. View the list of projects associated with each level-2 quota.

Metrics

The list of quotas and associated projects shows which projects have set the corresponding level-2 quota as their default computing quota.

Procedure

1. Log on to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, click Resource Observation.

3. On the Resource Observation page, click the Storage Resource tab to view the total storage usage and storage distribution for the current day.

4. (Optional) Select a time range (default is 7d) and projects (default is all projects; you can select up to eight projects) to view the Storage Trend.

5. (Optional) In the Storage Details area, on the Project Details tab, select a date (default is the current day) to view the storage usage of each project.

6. (Optional) In the Storage Details area, on the Table/Partition Details tab, select a date (default is the current day) and a project to view the detailed table and partition storage usage in the project.

Metrics

MaxCompute does not support viewing storage usage by column. To estimate column-level storage, export the table data and calculate the usage for each column.

Procedure

1. Log on to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Workspace > Resource Observation.

3. On the Resource Observation page, click the Data Transmission Service tab.

4. Select a quota, project, time range, and aggregation algorithm to query the usage of each metric.

Usage notes

Metrics

Procedure

1. Log on to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, click Resource Observation.

3. On the Resource Observation page, click the Open Storage tab.

4. Select the project, table, task initiator, and time range to query the usage of each metric.

Metrics

Procedure

1. Log on to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, click Resource Observation.

3. On the Resource Observation page, click the Job Performance Observation tab.

4. Select the following parameters to filter and group jobs. This allows you to view specific jobs and group metric data in charts by different dimensions.

   Parameter	Description
   Time range	Required. Filters completed jobs based on the specified time range (start and end times). You can select a preset time range or configure a custom one: 1d: The last 1 day. 3d: The last 3 days. 7d: The last 7 days. Select a specific period: Click the time range drop-down list, select the desired date, and then click Select Time to choose the target time period. Note The default time range is the last day. The maximum time range is 7 days, and the minimum is 1 hour. You can search for jobs from the last 45 days.
   Choose Project	Filters by MaxCompute project name. Note By default, all projects are selected. You can select up to 8 projects.
   Select Quota	Filters by computing quota. Note By default, all computing quotas are selected. You can select up to 8 level-2 quotas. For more information about computing quotas, see Manage computing quotas.
   Group By	Required. Depending on the chart type, you can group data in the chart view by multiple dimensions. Valid values for Group By: No Group (Default): Displays the trends of various metrics for all jobs within the filtered range over time. Project: Displays various metrics for all jobs within the filtered range, grouped by project. Note When you group by project, you must specify the projects in the filter parameters, up to a maximum of 8 projects. Quotas: Displays various metrics for all jobs within the filtered range, grouped by level-2 quota. Note When you group by quota, you must specify the quotas in the filter parameters, up to a maximum of 8 level-2 quotas. Job type: Displays various metrics for all jobs within the filtered range, grouped by job type. SQL: SQL job. MCQA: MaxCompute Query Acceleration (MCQA) SQL job. LOT: MapReduce job. CUPID: Spark or Mars job. Algo_Task: machine learning job. GRAPH: graph computing job. Job End Status: Displays various metrics for all jobs within the filtered range, grouped by their final status. Success: The job ran successfully. Failed: The job failed. Cancelled: The job was canceled.

5. Click Query to view the statistics for each metric.

6. (Optional) Select a Data Summary granularity to view the statistics for each metric based on the chosen time dimension.

   Parameter	Description
   By Hour	Summarizes data in one-hour intervals. This shows statistics for jobs that finished running in the current hour. This is the default setting. For example, if the current hour is 14:00 on May 6, 2024, summarizing by hour shows statistics for jobs that finished between 14:00 and 15:00 on May 6, 2024.
   By Day	Summarizes data in one-day intervals. This shows statistics for jobs that finished running on the current day. For example, if the current date is May 6, 2024, summarizing by day shows statistics for jobs that finished between 00:00 on May 6, 2024, and 00:00 on May 7, 2024.

7. (Optional) Select a Comparison Period to view historical statistics from a previous date or hour.

   The default is No Comparison. Options include Previous 30 Days, Previous 7 Days, and Previous 1 Day. For example, for 14:00 on May 6, 2024, the 30-day comparison data is the statistics from 14:00 on April 6, 2024.

Metrics

• CU Usage Trend (Unit: Core*H)

  Metric	Description
  CPU-hour (Unit: Core*H)	The number of CPU-hours consumed by completed jobs within the selected filter range. One CPU-hour is defined as one CPU core consumed for one hour. CPU-hours = Number of CPU cores × Duration.
  Memory-hour (Unit: GB*H)	The amount of memory-hours consumed by completed jobs within the selected filter range. One memory-hour is defined as 1 GB of memory consumed for one hour. Memory-hours = Memory size × Duration.
  CPU-hour/Top 10 Memory-Hour Consumption Analysis	Ranks the top 10 jobs by CPU-hour or memory-hour consumption. It also lists the top 10 Signatures and ExtNodeIds with the highest total or average consumption within the selected filter range.

• Job Runtime Period (Unit: seconds)

  Metric	Description
  Average Value	The average runtime of completed jobs within the selected filter range.
  Maximum Value	The longest runtime of completed jobs within the selected filter range.
  Minimum Value	The shortest runtime of completed jobs within the selected filter range.
  Quantile	The job runtime for a specific quantile (including the 1st, 5th, 10th, 50th, 90th, 95th, and 99th percentiles) of completed jobs within the selected filter range. For example, the 99th percentile indicates that 99% of jobs finished within this runtime.
  Top 10 Job Runtime Analysis	Provides the top 10 jobs with the longest total runtime, and the top 10 Signatures and ExtNodeIds with the longest total or average runtime within the selected filter range.

• Job Count Trend (Unit: count): The number of completed jobs within the selected filter range.

• Job Input Size Trend (Unit: GB, automatically adjusted based on the chart's scale): The amount of data scanned by completed jobs within the selected filter range.

• Trend of Job Input Size per CU-Hour (Unit: GB, automatically adjusted based on the chart's scale): The average size of data scanned by jobs per CU-hour within the selected filter range. One CU-hour includes 1 CPU core and 4 GB of memory. It is calculated by using the formula MAX(CPU-hour, CEILING(memory-hour/4)).

You can also collect statistics for the preceding metrics by using the tenant-level Information Schema. Note that the Information Schema task_history table contains all task instances resulting from operations, whereas the metrics on the Job Performance Observation tab in the console count only jobs that consume computing resources. Therefore, the statistical results may differ.

The following is a sample query:

SET odps.namespace.schema=TRUE;
SELECT to_char (end_time, 'yyyy-mm-dd hh'), -- The hour when the job finished.
       -- to_char (end_time, 'yyyy-mm-dd'), -- For daily granularity, use this line to replace the preceding one. This indicates the date when the job finished.
       sum(cast(cost_cpu/100/3600 as DECIMAL(18,5) )) cost_cpuh, -- CPU-hour
       sum(cast(cost_mem/1024/3600 as DECIMAL(18,5) )) cost_memh, -- Memory-hour
       avg(datediff(end_time, start_time, 'ss')), -- Average job runtime
       min(datediff(end_time, start_time, 'ss')), -- Minimum job runtime
       max(datediff(end_time, start_time, 'ss'))  -- Maximum job runtime
       -- status, -- Grouping basis: status: job status; project: task_catalog; job type: task_type.
FROM SYSTEM_CATALOG.INFORMATION_SCHEMA.tasks_history
WHERE ds>=to_char(date_add(getdate(),-7),'yyyymmdd')  -- Modify or add other filter conditions as needed.
and task_type in ('SQL','SQLRT','LOT','CUPID','ALgoTask')
GROUP BY to_char (end_time, 'yyyy-mm-dd hh')
         -- to_char (end_time, 'yyyy-mm-dd'), -- For daily granularity, use this line to replace the preceding one. This indicates the date when the job finished.
         -- status, -- Grouping basis: status: job status; project: task_catalog; job type: task_type.
order BY to_char (end_time, 'yyyy-mm-dd hh') ASC;
         -- to_char (end_time, 'yyyy-mm-dd'); -- For daily granularity, use this line to replace the preceding one. This indicates the date when the job finished.

FAQ

• Question 1:

  • Q: Why are some projects or quotas missing from the chart legend after grouping?

  • A: This happens when the job count for a project or quota is zero within the selected time range.

• Question 2:

  • Q: Why is comparison data missing for a previous period?

  • A: Data may be missing if the project or quota did not exist or had no jobs during the comparison period.