Manage Ray clusters

Create a Ray cluster

1. Log in to the EMR Serverless Spark console and go to your target workspace.

2. In the left-side navigation pane, click Cluster Management, and then click the Ray clusters tab.

3. Click Create Ray cluster. In the panel that appears, configure the following parameters and click Create.

   • Cluster name: Enter a name for the cluster.

   • Engine version: Select an engine version. The current default is err-1.0.1 (Ray 2.47.1, Python 3.12).

   • Node groups: If you need to use a mix of CPU and GPU resources, you can create separate CPU and GPU node groups. Click + Add worker node group and configure the following parameters.

     • Node group name: Enter a unique name for the node group.

     • Resource queue: Select a resource queue.

     • Resource specification: Select a resource specification for the nodes.

     • Number of nodes: Set the number of worker nodes.

     Click the expand icon to the right of a node group to configure optional advanced settings:

     • Enable Fault Tolerance (head node group only): Requires an external Redis service. If the head node crashes unexpectedly, the cluster restarts the head process and recovers its previous state.

     • Custom image version: Different node groups can use different engine versions, but the Ray engine version must be the same across all node groups. Otherwise, the cluster fails to start. If left unconfigured, the engine version selected for the Ray cluster is used.

     • Environment variables: Custom environment variables for the node, one per line, in the format key1=value1.

     • Startup parameters: Custom startup parameters for the node, separated by spaces, such as --num-cpus=0.

   • (Optional) Network connection: If you need to access services within the same VPC from the Ray cluster, select an existing network connection.

   • (Optional) Mount managed file directory: Select an existing managed file directory to enable your Ray jobs to read from and write to files within that directory. Mounting is a common data access method that maps paths from CPFS, NAS, and OSS to a local directory, allowing your programs to access data using local paths. To create a new managed file directory, see Manage Managed File Directories.

   • (Optional) Advanced cluster configurations: Configure advanced parameters in JSON format. For more information, see Advanced cluster configurations.

Start the Ray cluster

1. In the list of Ray clusters, find your target cluster and click Start.

2. Wait for the cluster status to change to running.

   Note

   Starting a Ray cluster for the first time in a workspace creates additional components and can take 2 to 3 minutes. Subsequent starts are significantly faster.

3. After the cluster starts, click invocation information to view the endpoint addresses and token required for submitting Ray jobs.

   Warning

   In the current version, the token for a Ray cluster does not expire. Keep your token secure to prevent unauthorized access.

   Click Dashboard to open the Ray cluster's monitoring interface, where you can view cluster resource usage.

View historical cluster instances

After a Ray cluster is stopped, its historical instance information is still available. On the details page of the cluster, the historical cluster instance list below the Overview tab shows the instance ID, status, start time, and stop time of each instance.

Click Details in the Actions column of an instance to view its details, or click Dashboard to open the Ray Dashboard of that instance and view the data generated before the cluster stopped.

Dynamically edit worker node groups

While a Ray cluster is running, you can edit worker node groups:

• Add and delete worker node groups.

• Modify the maximum and minimum replica counts of existing worker node groups.

Send Ray cluster metrics to SLS

A Ray cluster can send its metrics to Simple Log Service (SLS) for monitoring. Before you start, complete RAM authorization and the advanced cluster configuration.

1. Add RAM permissions: Grant the workspace role the permission to write to the Logstore. The default workspace role is AliyunEMRSparkJobRunDefaultRole. Check the workspace details page for the actual role. Sample policy:

   {
     "Version": "1",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "log:PostLogStoreLogs"
         ],
         "Resource": [
           "acs:log:*:*:project/<your-project-name>/logstore/<your-logstore-name>"
         ]
       }
     ]
   }

2. Configure the advanced cluster parameters: Add the following JSON to the advanced cluster configurations when you create the Ray cluster:

   {
     "slsProject": "<your-project-name>",
     "slsLogStore": "<your-logstore-name>"
   }

After the cluster starts, metrics are pushed to the specified Logstore every 30 seconds. The log topic is emr-ray.

You can query the Ray cluster metrics on the Metrics Explorer page of the corresponding project in the SLS console by using PromQL. For example, to query the CPU utilization:

ray_node_cpu_utilization{podName="ray-xxxxx"}

By default, metrics carry labels such as rayClusterId, podName, and RayNodeType that you can use to filter the results.

Advanced cluster configurations

Specify advanced cluster configurations in JSON format. All parameters are optional.

What to do next

After the cluster is running, you can submit tasks to it.

You can submit tasks through interactive development, the SDK, or the command line. For more information, see Get started with Ray clusters.