Enterprises that use MaxCompute often create multiple projects for data isolation based on their business requirements. Because projects require varying amounts of computing resources (CUs), you must isolate these resources by project to maximize their utilization. This topic explains how to use the quota management feature in the MaxCompute console to isolate subscription CUs.
Background information
You can use the quota management feature in the MaxCompute console to isolate purchased subscription CUs. The Quotas feature supports the following types of quota groups:
-
default subscription quota: This quota group is automatically created when you purchase subscription computing resources. You cannot modify this quota group. When you upgrade or downgrade your resources, the number of CUs in this quota group increases or decreases accordingly.
-
custom quota group: For a custom quota group, the Reserved CUs [minCU,maxCU] value is deducted from the default subscription quota group. The maxCU can be set to the total number of purchased reserved CUs. The parameters are as follows:
-
minCU: The minimum guaranteed number of CUs.
-
maxCU: The maximum number of CUs that the quota group can use. If you configure multiple quota groups with minCU < maxCU, a quota group can borrow idle CUs from other groups. This allows all quota groups in the current account and region to share a resource pool.
-
The sum of the minCU values across all quota groups must equal the total number of purchased reserved CUs.
Use case
Scenario
A data warehouse team uses MaxCompute for various business activities, such as data warehouse development and production, operational analysis, and algorithm mining. To support these activities, the team creates different MaxCompute projects:
-
Data warehouse development and production: Projects are divided into development and production environments based on the layers of the data warehouse model.
-
Operational analysis: Dedicated projects are created for different departments to retrieve and analyze daily operational data.
-
Algorithm mining: Projects are separated into development and production based on the characteristics of their job cycles.
Based on an initial business assessment, the team purchases a total of 1,000 reserved CUs. Now, the team needs to allocate and isolate these CUs effectively to maximize utilization.
Allocate CUs
When you allocate CUs, consider the following:
-
Configure the minCU for Reserved CUs based on actual requirements to prevent resource abuse.
-
For low-priority projects that do not require high availability but might request large amounts of CUs, limit the maxCU for the Reserved CUs in the corresponding quota group to avoid affecting other quota groups.
-
For MaxCompute projects with long-running jobs, create a dedicated quota group and limit the maxCU for Reserved CUs in that quota group.
-
You can use the time-specific configuration feature to maximize CU utilization. For more information, see Computing Resources - Quota Management.
-
The CU quota of the default subscription quota group cannot be modified. To prevent jobs from specific projects from consuming all available CUs, do not associate any projects with the default subscription quota group. The minCU for the Reserved CUs of the default subscription quota group cannot be 0. You can reserve 1 CU for this group and set minCU < maxCU for other quota groups. This allows other quota groups to use this 1 CU.
Design quota groups
|
Quota group |
Characteristics |
Reserved minCU |
Reserved maxCU |
Associated project |
Time configuration |
|
Data warehouse production quota group |
High-guarantee workload with significant peak-hour CU demand and stable hourly CU demand. |
|
|
Data warehouse production project |
|
|
Data warehouse development quota group |
High-guarantee workload with significant off-peak CU demand. |
|
|
Data warehouse development project |
|
|
Algorithm production quota group |
High-guarantee workload with some long-running, resource-intensive jobs. |
300 |
300 |
Algorithm production project |
00:00:00–23:59:59 |
|
Algorithm development quota group |
High-guarantee workload with some long-running, resource-intensive jobs. |
50 |
50 |
Algorithm development project |
00:00:00–23:59:59 |
|
Operational analysis quota group |
Demand peaks during business hours and includes occasional long-running, resource-intensive jobs. |
|
|
Operational analysis project |
|
Given these business characteristics, the maxCU value for the reserved CUs of each quota group is capped to prevent it from affecting the minimum guarantees of other groups. In this design, the default subscription quota group is not associated with any MaxCompute project. Because its minCU for Reserved CUs must be greater than zero, we set it to 1 CU. To balance the total, we reduce the minCU of the data warehouse development project by 1.
Procedure
-
Configure time periods. Add two periods, 00:00:00–09:00:00 and 09:00:00–23:59:59. For more information, see Computing Resources - Quota Management.
-
Based on your plan, create new quota groups: the data warehouse production, data warehouse development, algorithm production, algorithm development, and operational analysis quota groups. For more information, see Configure Quotas.
-
Associate your projects with the corresponding quota groups. The default subscription quota created upon purchase is not automatically associated with any project and must be associated manually.
Note-
To associate a quota group at the project level, see Manage projects. On the Parameter Configuration tab, configure the Default Compute Quota for the target project.
-
To associate a quota group at the job level, see Quota Rules and configure a quota for the target job.
-
References
-
Your quota group allocations may need to change as your business evolves. Therefore, monitor quota group usage regularly to make timely adjustments. You can use the MaxCompute subscription quota group resources metric in CloudMonitor to monitor usage. For more information, see monitoring and alerts.
-
You can also use other MaxCompute resource management features for more fine-grained control, such as Elastically Reserved CUs Best Practices, Use Quotas for Computing Resources, and job priority.
-
For best practices on time-specific quota configuration for subscription resources, see Time-specific Quota Configuration for Subscription Resources.
-
To learn how to query the usage of your computing resources (CUs), see resource observation.