Submit a Ray job

Prepare a node image

For each job, DLC uses a specified node image to create containers for the head and worker nodes, automatically building the cluster. When the cluster is ready, DLC starts a submitter node to submit your job to the cluster. The submitter node also uses the same image.

The Ray image must be version 2.6 or later and include at least the ray[default] components. The following images are supported:

• PAI provides official images with pre-installed Ray components.

• Ray community images:

  • We recommend using the rayproject/ray Docker image.

  • You can also use the rayproject/ray-ml image, which comes with built-in machine learning frameworks such as PyTorch and TensorFlow.

  GPU jobs require a CUDA-enabled image. For more information about supported image versions, see the official Docker image documentation.

Prepare a startup command and script file

The startup command for a DLC job is used as the entrypoint command for ray job submit. The startup command can be a single-line or multi-line command, such as python /root/code/sample.py, where:

• sample.py is the Python script to run. Mount the script file into the DLC container using a dataset or Code Builds. The following code is an example:

  import ray
  import os
  
  ray.init()
  
  @ray.remote
  class Counter:
      def __init__(self):
          # Used to verify runtimeEnv
          self.name = os.getenv("counter_name")
          # assert self.name == "ray"
          self.counter = 0
  
      def inc(self):
          self.counter += 1
  
      def get_counter(self):
          return "{} got {}".format(self.name, self.counter)
  
  counter = Counter.remote()
  
  for _ in range(50000):
      ray.get(counter.inc.remote())
      print(ray.get(counter.get_counter.remote()))

• /root/code/ is the mount path.

Use the console

1. Go to the Create Job page.

   1. Log on to the PAI console. In the top navigation bar, select a region. In the upper-right corner, select a workspace, and click Go to DLC.

   2. On the Deep Learning Containers (DLC) page, click Create Task.

2. On the Create Job page, configure the following key parameters. For details about the other parameters, see Create a Training Job.

   Parameter		Description	Example
   Environment Information	Node Image	On the Alibaba Cloud Image tab, select the pre-built official Ray image.	ray:2.39.0-cpu-py312-ubuntu22.04
   	Startup Command	The command to run for this job.	python /root/code/sample.py
   	Third-party Libraries	You can configure dependencies for Ray's runtime environment (runtime_env) by providing a list of third-party libraries. Note In a production environment, use pre-built images to run jobs. This avoids failures caused by on-the-fly dependency installations.	Not required for this example.
   	Code Builds	You can upload the prepared script file to the DLC container by using Online configuration or Local Upload.	To use the Local Upload method: Example code file: sample.py Mount path: /root/code/
   Resource Information	Source	Select Public Resources or Resource Quota.	Public Resources
   	Framework	The type of the framework.	Ray
   	Job Resource	Number of job nodes: A Ray cluster can be configured with head and worker node types. When configuring resources, the number of head nodes must be 1. The head node only runs the entrypoint script and does not act as a Ray worker node. A job typically requires at least one worker node, but this is not mandatory. Each Ray job automatically generates a submitter node to execute the startup command, and you can view the job logs in the submitter's logs. In pay-as-you-go jobs, a node of the smallest available resource type is generated, while in subscription jobs, the submitter node shares a small amount of user resources. Number of resources: The Logical Resources on a worker node in a Ray cluster match the physical resources that you configure when you submit a job. For example, if you configure a node with 8 GPUs, the default resource size of the worker node is also 8 GPUs. Ensure that the resource configuration meets your job's requirements. We recommend using a few large nodes instead of many small ones. Each node should have at least 2 GiB of memory. Increase the memory as the number of Tasks or Actors grows to avoid out-of-memory (OOM) errors.	Number of nodes: 1 for both. Instance type: Select ecs.g6.xlarge.

3. Worker node scaling. When you use a Resource Quota, click the icon to scale the number of worker nodes. You set the minimum and maximum number of instances for a role, and the system dynamically adjusts the actual number of running instances based on resource availability and load. This ensures that multiple tasks can run and optimizes global task execution and resource utilization.

   

   Click the auto scaling configuration, specify the number of roles, GPU, CPU, and memory size, and select an appropriate Scaling Policy. The supported scaling policies are:

   • Default-Maximize Nodes: The system maximizes the number of nodes when sufficient resources are available.

   • RayAutoscaler-Dynamic Scaling: Specific to Ray jobs, this policy automatically adjusts the node count based on the cluster workload.

   • CloudMonitorMetric-CloudMonitor Metrics: The system automatically adjusts the node count based on CloudMonitor Metrics (you can configure target values for CPU utilization, memory utilization, GPU compute utilization, and GPU memory utilization).

4. Log persistence is supported. When you use Public Resources, click Ray Log Collection to configure an OSS path for log storage. You can persist Ray Job logs, including Ray cluster logs, framework logs, and task logs, to a custom storage instance to facilitate subsequent debugging and analysis of Ray tasks.

5. After you configure the parameters, click OK.

Use the SDK

1. Install the Python SDK for DLC.

   pip install alibabacloud_pai_dlc20201203==1.4.0

2. Submit a DLC Ray job. The following code provides an example:

   #!/usr/bin/env python3
   
   from alibabacloud_tea_openapi.models import Config
   from alibabacloud_credentials.client import Client as CredClient
   
   from alibabacloud_pai_dlc20201203.client import Client as DLCClient
   from alibabacloud_pai_dlc20201203.models import CreateJobRequest
   
   region_id = '<region-id>'
   cred = CredClient()
   workspace_id = '12****'
   
   dlc_client = DLCClient(
       Config(credential=cred,
              region_id=region_id,
              endpoint='pai-dlc.{}.aliyuncs.com'.format(region_id),
              protocol='http'))
   
   create_job_resp = dlc_client.create_job(CreateJobRequest().from_map({
       'WorkspaceId': workspace_id,
       'DisplayName': 'dlc-ray-job',
       'JobType': 'RayJob',
       'JobSpecs': [
           {
               "Type": "Head",
               "Image": "dsw-registry-vpc.<region-id>.cr.aliyuncs.com/pai/ray:2.39.0-gpu-py312-cu118-ubuntu22.04",
               "PodCount": 1,
               "EcsSpec": 'ecs.c6.large',
           },
           {
               "Type": "Worker",
               "Image": "dsw-registry-vpc.<region-id>.cr.aliyuncs.com/pai/ray:2.39.0-gpu-py312-cu118-ubuntu22.04",
               "PodCount": 1,
               "EcsSpec": 'ecs.c6.large',
           },
       ],
       "UserCommand": "echo 'Prepare your ray job entrypoint here' && sleep 1800 && echo 'DONE'",
   }))
   job_id = create_job_resp.body.job_id
   print(f'jobId is {job_id}')

   Parameters:

   • region_id: The ID of the Alibaba Cloud region. For example, cn-hangzhou is the ID for the China (Hangzhou) region.

   • workspace_id: The ID of your workspace. You can find this ID on the workspace details page. For more information, see Manage workspaces.

   • Image: Replace <region-id> with the actual Alibaba Cloud region ID. For example, the region ID for China (Hangzhou) is cn-hangzhou.

For more information about how to use the SDK, see Python SDK.

Use the CLI

1. Download the DLC client tool and complete user authentication. For more information, see Before you begin.

2. Submit a DLC Ray job. The following code provides an example:

   ./dlc submit rayjob --name=my_ray_job \
     --workers=1 \
     --worker_spec=ecs.g6.xlarge \
     --worker_image=dsw-registry-vpc.<region-id>.cr.aliyuncs.com/pai/ray:2.39.0-cpu-py312-ubuntu22.04 \
     --heads=1 \
     --head_image=dsw-registry-vpc.<region-id>.cr.aliyuncs.com/pai/ray:2.39.0-cpu-py312-ubuntu22.04 \
     --head_spec=ecs.g6.xlarge \
     --command="echo 'Prepare your ray job entrypoint here' && sleep 1800 && echo 'DONE'" \
     --workspace_id=4****

   For more information about how to configure parameters when you submit a job by using the CLI, see Submit command.

Q: Ray job fails during environment preparation

• Check the head node log to verify that the Ray environment started correctly. If not, Ray is unavailable in the instance. Prepare an image that supports Ray as described in the Before you begin section.

  

• Check the head node event log. If you see a Readiness probe failed... error, the image might be missing dependencies for the readiness check, or some indirect dependencies might be unavailable. Reinstall the ray[default] components in the original image by using pip or conda, or try to rebuild the image based on an official Ray image.