Advanced notebook development

Reference project resources (.py files)

Encapsulate common functions or classes into separate .py files and reference them as MaxCompute resources by using ##@resource_reference{"custom_name.py"}. This approach modularizes your code, improves reusability, and simplifies maintenance.

1. Create and publish a Python resource

   1. In the left-side navigation pane of DataWorks DataStudio, click the icon to go to Resource Management.

   2. In the Resource Management tree, right-click the target directory or click + in the upper-right corner. Select New Resource > MaxCompute Python, and name the resource my_utils.py.

   3. In the Document Content section, click Online Editing, paste your debugged utility function code into the editor, and click Save.

      # my_utils.py
      def greet(name):
          return f"Hello, {name} from resource file!"

   4. In the toolbar, click Save and then Publish. This makes the resource visible to both development and production tasks.

2. Reference the resource in a notebook

   In the first line of a Python cell in your notebook, use the ##@resource_reference syntax to reference the published resource.

   ##@resource_reference{"my_utils.py"}
   # If the resource is in a directory, such as my_folder/my_utils.py, you still use ##@resource_reference{"my_utils.py"} without the directory name.
   from my_utils import greet
   message = greet('DataWorks')
   print(message)

3. Debug and run in the development environment

   Run the Python cell. The following result is printed:

   Hello, DataWorks from resource file!

   Important

   When you debug and run in the development environment, the system detects the ##@resource_reference declaration and automatically downloads the target file from Resource Management to the workspace/_dataworks/resource_references path in your personal directory. This allows you to reference the file.

   If a ModuleNotFoundError error occurs, click the Restart button in the editor toolbar to reload the resource, and then try again.

4. Publish to the production environment and verify

   After you Save and Publish this Notebook node, go to Operation and Maintenance Center > Auto Triggered Task and click Test to run it. After the task runs successfully, you will see the output Hello, DataWorks from resource file! in the logs.

   Important

   If a There is no file with id ... error occurs, ensure you have published the Python resource to the production environment first.

For more operations, see MaxCompute resources and functions.

Read and write datasets (OSS/NAS)

Read and write large files stored on OSS or NAS while a notebook task is running.

For more operations, see Use datasets in a personal development environment.

Reference workspace parameters

DataWorks introduces Workspace Parameters in addition to the existing scheduling parameters to reuse global configurations and isolate environments across different tasks and nodes. You can reference Workspace Parameters in SQL and Python cells by using the format ${workspace.param}, where param is the name of the Workspace Parameter you created.

1. Create Workspace Parameters: In DataWorks, go to Operation and Maintenance Center > Scheduling Settings > Workspace Parameters to create the parameters.

2. Reference Workspace Parameters:

• Reference a Workspace Parameter in an SQL cell.

  SELECT '${workspace.param}';

  When the query runs successfully, it prints the resolved value of the parameter.

• Reference a Workspace Parameter in a Python cell.

  print('${workspace.param}')

  When the code runs successfully, it prints the resolved value of the parameter.

For more details, see Use workspace parameters.

Connect to MaxCompute

• %odps: Get a PyODPS entry object

  This is the recommended way to interact with MaxCompute because it avoids hard-coding an AccessKey in your code.

  1. Use a magic command to create a MaxCompute connection. Enter %odps. A MaxCompute compute resource selector appears in the lower-right corner (and automatically selects a compute resource). You can click the MaxCompute project name in the lower-right corner to switch MaxCompute projects.

     o=%odps 

  2. Use the returned MaxCompute compute resource to run a PyODPS script.

     For example, to get all tables in the current project:

     with o.execute_sql('show tables').open_reader() as reader:
         print(reader.raw)

• %maxframe: Establish a MaxFrame connection

  This command creates a MaxFrame session, which provides pandas-like distributed data processing capabilities for MaxCompute.

  # Connect to and access the MaxCompute MaxFrame session.
  mf_session = %maxframe
  df = mf_session.read_odps_table('your_mc_table')
  print(df.head())
  # After development and debugging, destroy the session to release resources.
  mf_session.destroy()

Connect to Spark compute resources

DataWorks Notebook supports connections to multiple Spark engines. These engines differ in connection methods, execution contexts, and resource management.

Connect to a Lindorm Ray compute resource

The Ray resource group of the Lindorm compute engine provides distributed computing services that support end-to-end AI workloads. Using a magic command, you can seamlessly connect to Lindorm Ray resources in a notebook for interactive development and debugging, and then publish the notebook as a production scheduling task.

1. Establish a connection: Run the %lindorm_ray magic command in a Python cell. After you run the command, a compute resource selector appears in the lower-right corner of the cell. Select your Lindorm compute resource and the Ray resource group you created.

   # Connect to the specified Lindorm Ray resource group.
   %lindorm_ray

   Important

   • After you connect to a Lindorm Ray compute resource, you can no longer run SQL cells in the same notebook. The Lindorm Ray engine focuses on running Python and Ray code.

   • If you run the same code cell multiple times, the system automatically terminates the previous Ray job and starts a new one. This prevents resource waste and task conflicts.

2. Execute Ray code: After a successful connection, you can directly write and run Ray code in a new Python cell. Logs are streamed to the output area of the cell in real time for interactive debugging.

   The following example defines a simple remote task using the @ray.remote decorator. The task runs on the Ray cluster, and the logs and final result are returned to the output area of the cell.

   import ray
   import time
   @ray.remote
   def hello_world():
     print("Hello from Lindorm Ray!")
     time.sleep(5)
     return "Task finished."
   # Submit the remote task.
   result_ref = hello_world.remote()
   print(ray.get(result_ref))

3. (Optional) Specify custom startup parameters: If you need to specify additional configurations for the Ray environment, such as installing third-party Python packages or uploading local code files, use the %%lindorm_ray command to establish the connection.

   • Example 1: Install dependencies

     Use the pip parameter to install the jieba package in the Ray environment.

     %%lindorm_ray
     {
       "runtime_env": {
         "pip": ["jieba"]
       }
     }

     After the environment is ready, import and use the package in subsequent Ray tasks. The following example shows how to call jieba in a remote function for Chinese word segmentation:

     import ray 
     @ray.remote
     def do_work(x):
         import jieba
         return "/".join(jieba.cut(x))
     print(ray.get(do_work.remote("欢迎使用DataWorks+LindormRay解决方案")))

   • Example 2: Upload and use DataWorks resources

     Use the working_dir parameter to upload resources from Resource Management to the Ray cluster, so they can be imported and called in your tasks.

     Important

     • When you use working_dir to upload resources, the files are uploaded directly from your development environment to the Ray cluster. A size limit of 100 MB applies. If the resource package is too large, the upload may fail or the Ray node may become unstable.

     • For large resource files or dependencies (over 100 MB), we recommend that you first upload them to OSS and then download and read them from OSS in your code, or package them directly into a custom image. This provides better stability and performance.

     # Reference and declare the path for a resource from Resource Management.
     %%lindorm_ray
     {
         "runtime_env": {
             "working_dir": "/mnt/workspace/_dataworks/resource_references"
         }
     }

     Assume that you have uploaded a file named ray_resource.py to Resource Management in DataWorks. When you write and run the following cell, the system automatically parses the ##@resource_reference declaration in the subsequent code and downloads the corresponding resource to the /mnt/workspace/_dataworks/resource_references path.

     Example code in ray_resource.py:

     def fun():
         print("This is a test function in ray_resource.py")

     Important

     In the development environment, after you run the cell that contains ##@resource_reference, you must rerun the preceding %%lindorm_ray cell. Rerunning this cell includes the downloaded resources in the working_dir and uploads them to the Ray cluster. You do not need to rerun the cell in the production environment.

     import ray 
     ##@resource_reference{"ray_resource.py"}
     @ray.remote
     def do_work(x):
         print('Ray says:', x)
         from ray_resource import fun
         fun()
         return x
     worker = do_work.remote("欢迎使用DataWorks+LindormRay解决方案")
     print(ray.get(worker))

4. Production scheduling and O&M: After development and debugging, you can commit and publish this notebook node. It will be scheduled periodically as a Lindorm Ray node in a DAG.

   • Parameterization: Your code can use standard DataWorks scheduling parameters, such as ${bizdate}.

   • Log viewing: In a production environment, to prevent excessive logs from affecting performance, the system loads only the first 1 MB of logs by default. If the logs are truncated, the output provides a link that you can click to view the complete task logs in the Lindorm console.

   • Resource release: After a scheduled production task ends, the Lindorm Ray task enters a terminal state and no longer occupies resources. During interactive development, you can terminate the Lindorm Ray task by restarting the kernel or closing the notebook.

Appendix: Magic command quick reference