Overview of custom scheduling resource groups
Dataphin lets you manage scheduling resources by grouping them. You can partition resources into groups, each with a distinct quota, and these quotas are independent of each other. You can then assign specific resource groups to tasks under different projects. This approach prevents core tasks from being preempted and improves overall resource utilization.
Learn in 5 minutes
Permissions
Super administrators, system administrators, and resource group owners can manage custom resource groups. This includes editing, enabling, disabling, and deleting them.
Scenarios
Protect core tasks: You can manage resource groups to ensure that core tasks run with priority. This prevents low-priority tasks from preempting large amounts of scheduling resources, which would cause high-priority tasks to wait for resources.
Allocate resources as needed: Resource groups support resource allocation by project. This lets you decouple resources used by different business departments and enables scenarios such as pay-as-you-go billing for each department.
Terms
Scheduling resources: These are the resources that task instances consume during scheduling. You can assign all available scheduling resources for the current to different resource groups and associate them with projects. Resources are isolated between resource groups.
default resource group: Each instance has a single default resource group. The system automatically creates this group when the instance is created, and it cannot be deleted. When you create a custom resource group from the public scheduling cluster , resources are automatically allocated from the default resource group. To ensure that all tasks have available scheduling resources, you must set a minimum percentage of resources to reserve for the default resource group. This ensures that the default resource group can be used as a fallback if a specified resource group is unavailable during scheduling, which prevents disruptions to task execution.
Custom resource group: Within instance, you can create multiple custom resource groups. You can set application scenarios for these groups to better control resource usage. Scenarios include daily task scheduling and batch O&M operations.
Daily task scheduling: These custom resource groups can be used for scenarios such as ad hoc queries, temporary runs, and recurring schedules. You must configure which projects can use the resource group.
Batch O&M operations: These resource groups can be used to temporarily specify a resource group for a single scheduling run, such as performing data backfills for production tasks.
Resource group priority: The resource group configured for a task is used first. If the custom resource group specified for a task instance is disabled or deleted when the instance is generated, the instance automatically uses the default resource group of its project. If the project's default resource group is also unavailable, it uses the default resource group. The priority is as follows: Task-specified resource group > Default resource group of the task's project > default resource group. Additionally, if the resource group specified for an instance is disabled or deleted at the time of scheduling, the instance automatically uses the default resource group for execution.
Process overview
After you purchase a Dataphin instance, the system calculates the total available resources for your public scheduling cluster based on the specifications of the purchased Data Processing Units (DPUs). For more information, see Purchasing guide for fully managed Dataphin instances.
Before you create a custom resource group, you must assign a minimum reserved percentage to the default resource group. The amount of resources allocated to the default resource group is dynamically adjusted based on the remaining resources of the default scheduling cluster, but it will not fall below the minimum reserved value. When you create a custom resource group in the default scheduling cluster, the corresponding percentage is partitioned from the tenant default resource group. For more information, see Introduction to the Resource Group Configuration page.
You can create multiple custom resource groups in the default scheduling cluster based on your resource management needs. You must specify the scheduling cluster for each custom resource group and the percentage of the cluster's available resources it can use. When you create a custom resource group, the required resource percentage is automatically deducted from the default resource group in the public scheduling cluster.
Resource group priority
When a task is scheduled, the resource group specified in the task's scheduling configuration is used first. If the specified resource group is in an abnormal state or has been deleted, the default resource group of the task's project is used. If the project's default resource group is also unavailable, the system uses the default resource group as a fallback. Additionally, to implement emergency throttling, you can change the resource allocation of a specific resource group to 0. The task will then remain in the Waiting for scheduling resources state until the resource group has available resources.
Task resource group configuration
You can configure scheduling resource groups for both auto triggered tasks and one-time tasks. For projects in Basic mode, test runs, smoke tests, and recurring schedules all use the configured scheduling resource group. For projects in Dev-Prod mode, test runs, smoke tests, and development environment schedules use the project's default resource group configured in the development project by default. Recurring schedules for production tasks use the resource group specified for each task by default.
Accessing resource group settings
On the Dataphin home page, click Management Center > System Settings in the top menu bar.
Go to the Resource Settings page.
Click Resource Settings > Resource Group Configuration.
