Interactive Ray development with DMS Notebook

Overview

As data volumes grow and machine learning applications become more widespread, data teams face the core challenge of efficiently training models, running distributed inference, and performing feature engineering on massive datasets. Traditional solutions often require users to move data between databases and compute clusters. This process not only introduces significant data transfer latency and storage costs but also requires teams to manage both database operations and distributed computing frameworks, making implementation more challenging.

To address this, AnalyticDB for MySQL integrates the Ray distributed computing framework with its existing Online Analytical Processing (OLAP) capabilities by introducing a built-in Ray resource group. You no longer need to set up and maintain a separate Ray cluster. Creating a Ray resource group in the AnalyticDB for MySQL console provides a distributed computing environment that is natively connected to your database storage and compute resources. This approach brings compute to the data, eliminating the bottleneck of cross-system data movement.

This topic provides a complete guide on how to connect to a remote Ray cluster hosted by AnalyticDB for MySQL from DMS Notebook. It demonstrates how to use Ray with three typical programming paradigms:

• Zero-code-intrusive parallelization with ray.util.multiprocessing.

• Asynchronous task orchestration with Remote Functions.

• Stateful service management with Actors.

Whether you want to seamlessly migrate existing Python scripts to a distributed environment or build production-scale machine learning pipelines, this guide provides clear, reproducible instructions.

Key advantages

Interactive Ray programming in a notebook offers the following advantages:

• Acts as a "remote control": Your Python script runs in the notebook and sends commands to the remote cluster over the network.

• Ideal for interactive debugging, exploratory analysis, and small-scale task distribution.

• Tasks submitted from your script run directly within the cluster, enabling efficient, low-latency communication between distributed workers.

Prerequisites

• You have created a Ray service in your AnalyticDB for MySQL cluster.

• You have created a notebook file in DMS Notebook. For instructions, see Steps 2 to 5 in Stream data by using PySpark.

  When you create a session, you must select a runtime image. The image determines the Python version of the notebook. Ensure that the Python version of the selected image matches that of the Ray cluster, which is 3.11.13.

• You have added the CIDR block of the VPC where the notebook resides to the IP whitelist of your AnalyticDB for MySQL cluster.

Procedure

from ray.util.multiprocessing import Pool

def square(x):
    return x * x

# The usage is identical to Python's native multiprocessing. The @ray.remote decorator is not needed.
with Pool() as pool:
    # This line automatically distributes the execution to the remote cluster.
    results = pool.map(square, range(10))
    print(f"Pool results: {results}")

@ray.remote
def add(x, y):
    return x + y

# asynchronous call - immediately returns an ObjectRef, which is similar to a Promise or Future.
ref1 = add.remote(1, 2)
ref2 = add.remote(3, 4)

# Block only when the actual result is needed.
final_result = ray.get(add.remote(ref1, ref2)) 
print(f"Task result: {final_result}")

@ray.remote
class Counter:
    def __init__(self):
        self.value = 0
    def increment(self):
        self.value += 1
        return self.value

# Create a persistent "counter" object on the cluster.
counter_actor = Counter.remote()

# Call the same Actor instance multiple times. The state is preserved.
results = [counter_actor.increment.remote() for _ in range(3)]
print(f"Actor counter results: {ray.get(results)}") # Output: [1, 2, 3]