Map baseline priorities to YARN queue priorities

Background information

YARN is a distributed resource management system that manages and schedules resources in an E-MapReduce (EMR) cluster. It allocates resources to the various jobs that run on it. Within YARN, the priority of a job's YARN queue determines which job receives resources first, controlling the scheduling order. For more information about YARN, see YARN Overview.

In DataWorks, you can configure the YARN queue for scheduled node tasks in one of the following ways:

• Method 1 (Global configuration): In the management center of a workspace, you can specify the YARN queue that a DataWorks module uses to run EMR tasks at the workspace level. For more information, see Configure global YARN resource queues.

• Method 2 (Node-specific configuration): On the configuration page of an EMR node, you can set the YARN queue and queue priority for that specific node. For more information, see Create an EMR Hive node, Create an EMR Spark node, and Create an EMR Spark SQL node.

If the 'Prioritize Global Configuration' option is enabled for the workspace, the globally configured YARN queue is used when scheduling the EMR node. Otherwise, the YARN queue configured on the node is used.

The final YARN queue priority that takes effect for a scheduled EMR job is determined by the following rules:

• If a node's baseline is enabled and a priority mapping is configured, the mapping determines the final YARN queue priority. The system uses this mapping to find the YARN queue priority that corresponds to the node's baseline priority.

  Note

  You can set the EMR node parameter ENABLE_TASK_PRIORITY=true on the node configuration page in Data Development. If this parameter is set to true, the node's configured priority overrides the priority derived from the baseline mapping.

• If no baseline priority mapping is configured, the priority specified on the EMR node is used.

Prerequisites

You have created an EMR node job and configured its YARN queue priority. For more information, see Create an EMR Hive node, Create an EMR Spark node, or Create an EMR Spark SQL node.

Limits

The following limits apply when you map baseline priorities to YARN queue priorities:

• Feature

  • This feature is supported only for EMR Hive, EMR Spark, and EMR Spark SQL nodes.

  • You must configure the maximum priority for the YARN queue in your EMR cluster before you can configure the baseline priority mapping.

  • To modify a YARN queue priority, go to the EMR console. You must restart YARN for the change to take effect. The modified priority takes effect only for the specified queue.

  Note

  For more information about how to configure YARN queue priorities, see DataLake cluster configurations.

• Permissions

  • Only a Tenant Administrator can configure the priority mapping. To grant this permission to another user, assign the Tenant Administrator role to them. For more information, see Manage the roles and permissions of tenant members.

  • This feature is configured at the tenant level. The mapping applies to the entire tenant.

  • Only users who have the AliyunDataWorksAccessingEMRReadOnlyPolicy policy can configure the mapping. For more information about how to grant permissions, see Grant permissions to a RAM user.

• Resource groups

  Exclusive resource groups for scheduling purchased before August 31, 2023 do not support this feature. If you use one of these older resource groups, contact technical support for an upgrade. If the resource group is not upgraded, the configured mapping does not take effect, and the system falls back to using the YARN queue priority configured on the node.

Access the configuration page

1. Go to the Operation Center page.

   Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Development and O&M > Operation Center. On the page that appears, select the desired workspace from the drop-down list and click Go to Operation Center.

2. In the left-side navigation pane, click Intelligent Baselines to go to the Smart Baseline configuration page.

Configuration logic

You can configure the mapping on the Intelligent Baselines > Priority Mappings Between Baselines and YARN Queues tab.

Based on your business requirements, select the cluster and YARN queue that run your EMR jobs, and then configure the mapping between the baseline priority and the YARN queue priority. The configuration follows these rules:

• The configured YARN queue priority cannot exceed the highest priority allowed for the YARN queue in the EMR cluster.

• For YARN queues, a higher number indicates a higher priority.

• A baseline with a lower priority must be mapped to a YARN queue priority that is less than or equal to that of a baseline with a higher priority.

For more information about YARN configurations, see YARN.