Overview of custom scheduling resource groups
Dataphin lets you manage scheduling resources by partitioning them into resource groups. Each resource group has an independent quota. You can assign specific resource groups to tasks within different tenants and projects. This method ensures that resources for core tasks are not preempted and improves resource utilization.
Quick start in 5 minutes
Limits
Currently, the custom resource group feature is available only for customers with deployments based on the latest architecture. For more information, contact the product Operations and Maintenance (O&M) team.
By default, the default cluster can manage a maximum of 72 CPU cores for scheduling. To increase this limit, you can purchase more resources. For more information about the total amount of allocatable resources, see Scheduling resource dashboard.
Permissions
Super administrators, system administrators, and resource group owners can manage custom resource groups. This includes editing, enabling, disabling, and deleting them.
Scenarios
Protecting core tasks: Resource group management ensures that core tasks run with priority. This approach prevents low-priority tasks from preempting large amounts of scheduling resources, which would otherwise cause high-priority tasks to wait for resources.
On-demand allocation: Resource groups support resource allocation by project, which lets you decouple resources used by different business departments. This enables scenarios such as on-demand billing for each department.
Terms
Scheduling resource: A scheduling resource is consumed when scheduling task-generated instances. You can allocate all scheduling resources available to the current tenant into different resource groups and associate them with projects. Resources in different resource groups are isolated from each other.
Default resource group for a tenant: Each tenant has exactly one default resource group. The system automatically creates this group when the tenant is created, and the group cannot be deleted. When you create a custom resource group from the default cluster, resources are automatically allocated from the tenant's default resource group. To ensure that all tasks have scheduling resources available, you can set a minimum resource percentage to reserve for the tenant's default resource group. This setting ensures that if a specified resource group is unavailable during scheduling runtime, the tenant's default resource group can be used as a fallback to prevent disruptions to task execution.
Custom resource group: Within each tenant , you can partition resources into multiple custom resource groups. You can set application scenarios for these groups to better control the scope of resource usage. Application scenarios include daily task scheduling and batch O&M operations.
Daily task scheduling: Custom resource groups for this scenario can be used for ad hoc queries, temporary runs, and recurring schedules. You must configure the associated projects that can use the resource group.
Batch O&M operations: Resource groups for this scenario can be used to temporarily specify a resource group for a single scheduling instance when you perform operations, such as data backfill for production tasks.
Resource group priority: The resource group configured for a task is used with the highest priority. If the custom resource group specified for a task instance is disabled or deleted, the instance automatically switches to the default resource group of its project. If the project's default resource group is also unavailable, the instance switches to the tenant's default resource group. The priority order is as follows: Task-specified resource group > Project's default resource group > Tenant's default resource group. Additionally, if the resource group specified for a scheduled instance is disabled or deleted, the instance automatically switches to the tenant's default resource group for execution.
Process overview
After you deploy Dataphin, the system determines the total available resources for your default cluster. This value is based on the resources available for task scheduling in the default cluster and the available scheduling resources from your purchase information. The calculation is as follows:
Total available CPU = min(CPU resources available for scheduling in the cluster configuration, purchased CPU resources available for custom resource group configuration).
Total available memory = Memory resources available for scheduling in the cluster configuration.
If the custom resource group feature is enabled, you must assign a maximum percentage of the default cluster's scheduling resources to each tenant during tenant creation. For more information, see Tenant settings.
Within each tenant, you can create multiple custom resource groups based on your resource management needs. For each custom resource group, you must specify the scheduling cluster and the percentage of available cluster resources to allocate.
When you create a custom resource group based on a tenant's default cluster, the required resource percentage is automatically deducted from the tenant's default resource group.
When you create a resource group based on a custom cluster, the required resource percentage is automatically deducted from the custom cluster.
Resource group effective priority
During task scheduling runtime, the resource group specified in the task's scheduling configuration is used with the highest priority. If the specified resource group is in an abnormal state or has been deleted, the default resource group of the task's project is used. If the project's default resource group is also unavailable, the system switches to the tenant's default resource group as a fallback. Additionally, if you want to implement emergency traffic limiting by adjusting resource groups, you can change the resource allocation of a specified resource group to 0. The task will then remain in the Waiting for scheduling resource state until the resource group has available resources.
Task resource group configuration
Both recurring tasks and one-time tasks support the configuration of scheduling resource groups. For projects in Basic mode, test runs, smoke tests, and recurring schedules all use the configured scheduling resource group. For projects in Dev-Prod mode, test runs, smoke tests, and development environment scheduling use the project's default resource group by default, and this setting cannot be customized. Recurring schedules for production tasks use the resource group specified for each task by default.
Accessing the resource group configuration
In the top menu bar on the Dataphin home page, click Management Center > System Settings.
To go to the Resource Settings page, follow the steps below.
In the navigation pane on the left, choose Resource Settings > Resource Group Configuration.
