Manage resource queues

Updated at:

Each workspace includes two default queues, dev_queue and root_queue, for running jobs. If you need to isolate and manage resources, you can create additional resource queues. This topic describes how to manage resource queues.

Prerequisites

You have created a workspace. For more information, see Manage workspaces.

Create a queue

Note

You can isolate and manage resources by adding queues. If you do not add new resource queues, you can use the default dev_queue and root_queue to run jobs. If you purchase a subscription quota, the system automatically creates a subscription_queue queue.

  1. Go to the Resource Management page.

    1. Log on to the E-MapReduce console.

    2. In the navigation pane, choose EMR Serverless > Spark.

    3. On the Spark page, click the name of the target workspace.

    4. On the EMR Serverless Spark page, click Resource Management in the navigation pane.

  2. Select the CPU or GPU tab based on the type of queue that you want to create, and then click Create Queue on the Queues tab.

  3. In the Create Queue dialog box, configure the parameters based on the queue type, and then click OK.

    CPU

    On the CPU tab, configure the following parameters based on your workspace's billing method.

    Pay-as-you-go

    Parameter

    Description

    Queue Name

    The queue name must be unique within the workspace.

    Note

    You cannot change the Queue Name after the queue is created.

    Auto Maximum Concurrency (CU)

    Specifies the maximum number of compute units (CUs) that can run concurrently in the queue. Based on the configuration of your session instances, set this value to at least 3 for the selected deployment queue.

    The sum of the auto maximum concurrency of all queues cannot exceed the pay-as-you-go quota of the workspace. If the pay-as-you-go quota is insufficient, adjust the pay-as-you-go quota of the workspace first. For more information, see Manage workspaces.

    To adjust the concurrency limit for other queues, see Edit a queue.

    Application Environment

    • Development: Recommended for non-critical workloads, such as development and testing.

    • Production: Recommended for business-critical workloads that require stable, efficient, and secure service.

    You can select both Development and Production to share the queue between development and production environments.

    Description

    The remarks of the queue. The description can be up to 256 characters in length.

    Subscription

    Parameter

    Description

    Queue Name

    The queue name must be unique within the workspace.

    Note

    You cannot change the Queue Name after the queue is created.

    Fixed Resource Quota (CU)

    The maximum number of CUs you can allocate depends on the remaining subscription quota in the workspace. If the quota is insufficient, you can:

    Application Environment

    • Development: Recommended for non-critical workloads, such as development and testing.

    • Production: Recommended for business-critical workloads that require stable, efficient, and secure service.

    Hybrid Billing

    Parameter

    Description

    Queue Name

    The queue name must be unique within the workspace.

    Note

    You cannot change the Queue Name after the queue is created.

    Auto Maximum Concurrency (CU)

    Specifies the maximum number of compute units (CUs) that can run concurrently in the queue. Based on the configuration of your session instances, set this value to at least 3 for the selected deployment queue.

    To adjust the concurrency limit for other queues, see Edit a queue.

    Fixed Resource Quota (CU)

    The maximum number of CUs you can allocate is determined by the remaining subscription quota in the workspace. If the quota is insufficient, you can adjust the resource allocation of other queues.

    For more information, see Edit a queue.

    Application Environment

    • Development: Recommended for non-critical workloads, such as development and testing.

    • Production: Recommended for business-critical workloads that require stable, efficient, and secure service.

    GPU

    Switch to the GPU tab and configure the following parameters. After you purchase a subscription GPU quota for the first time, the system automatically creates a gpu_<spec>_queue queue. When you add a pay-as-you-go GPU quota for the first time, the system initializes the gpu_dev_queue and gpu_root_queue queues at a card ratio of 1:4.

    Parameter

    Description

    Queue Name

    The queue name must be unique within the workspace.

    Note

    You cannot change the queue name after the queue is created.

    Billing Type

    Pay-as-you-go or subscription.

    GPU (Cards) Maximum Concurrency

    The maximum number of GPU cards that the queue can use to concurrently process jobs.

    Instance Type

    Limit GPU instance types is enabled by default. When this option is enabled, only the GPU instance types within the selected range can be used.

    Application Environment

    • Development: Recommended for non-critical workloads, such as development and testing.

    • Production: Recommended for business-critical workloads that require stable, efficient, and secure service.

    You can select both Development and Production to share the queue between development and production environments.

    Description

    The remarks of the queue.

Edit a queue

CPU

On the CPU tab, find the target queue and click Edit Queue in the Actions column. On the Resources tab of the Edit Queue dialog box, you can adjust the queue resource configurations and modify the Application Environment and Description in addition. You cannot change the queue name. The resource configurations vary based on your workspace's billing method.

Pay-as-you-go

  1. On the Queues tab of the Resource Management page, find the target queue and click Edit Queue in the Actions column.

  2. In the Edit Queue dialog box, on the Resources tab, adjust the value for Auto Maximum Concurrency (CU).

  3. Click OK.

Subscription

  1. On the Queues tab of the Resource Management page, find the target queue and click Edit Queue in the Actions column.

  2. In the Edit Queue dialog box, on the Resources tab, adjust the value for Fixed Resource Quota (CU).

  3. Click OK.

Hybrid Billing

  1. On the Queues tab of the Resource Management page, find the target queue and click Edit Queue in the Actions column.

  2. In the Edit Queue dialog box, on the Resources tab, adjust the values for Fixed Resource Quota (CU) and Auto Maximum Concurrency (CU).

  3. Click OK.

GPU

Switch to the GPU tab, find the target queue, and click Edit Queue in the Actions column. On the Resource tab of the Edit Queue dialog box, you can adjust the GPU (Cards) Maximum Concurrency and Instance Type values, and modify the Application Environment and Description in addition. You cannot change the queue name.

Note

Rules for editing GPU queues:

  • Number of cards: the adjusted value must be greater than or equal to the number of GPU cards in use.

  • Instance types: you can only add instance types, not remove them.

Edit queue permissions

  1. On the Queues tab of the Resource Management page, find the target queue and click Edit Queue in the Actions column.

  2. In the Edit Queue dialog box, click the Permission tab.

  3. You can add or remove users, and modify their permissions.

  4. Click OK.

Delete a queue

Important
  • You cannot delete the root_queue.

  • Before you delete a resource queue, ensure that no running or pending jobs in the workspace are associated with it. Deleting a queue that is in use affects all associated jobs.

  1. Select the CPU or GPU tab based on the type of the target queue, find the target queue on the Queues tab, and then click Delete in the Actions column.

  2. In the Delete Queue dialog box, click Delete Queue.

References