Use MaxFrame in a DataWorks notebook

Prerequisites

Step 1: Activate MaxCompute

MaxCompute is a large-scale data warehousing and computing service from Alibaba Cloud that supports the MaxFrame distributed Python computing framework.

1. Log on to the MaxCompute console, and select a region in the upper-left corner.

2. In the navigation pane on the left, choose Manage Configurations > Projects.

3. On the Projects page, click Create Project to open the Create Project dialog box. Then, configure the project information as prompted and click OK.

   • Region: Must be the same as the region you will select for DataWorks, for example, China (Shanghai).

   • Project Name (Globally Unique): Enter a custom name. We recommend including a business identifier.

   • Billing Method:

     • Subscription: Suitable for long-term, stable high-load production workloads. You pre-purchase a fixed number of compute units (CUs) to guarantee resource availability and avoid resource contention.

     • Pay-as-you-go: (Recommended) Suitable for short-term or testing phases. This method offers flexible billing based on the actual CUs consumed.

Step 2: Activate DataWorks

DataWorks is an all-in-one big data development and governance platform that provides capabilities such as notebook development, job scheduling, data integration, and data governance.

1. Go to the DataWorks purchase page to activate a DataWorks edition.

   Parameter	Description	Recommended value
   DataWorks edition	Features and pricing vary by edition.	Basic Edition (Free, meets requirements for notebook development).
   Region and availability zone	Must be the same as the MaxCompute project region.	Select based on your business needs.

2. Go to the resource group purchase page to purchase compute resources for running tasks. Configure the following parameters:

   Parameter	Description
   Region and availability zone	Must be the same as the MaxCompute project region. Select based on your business needs.
   Billing method	pay-as-you-go (Recommended) subscription
   Resource group name	Enter a custom name. The name must start with a letter, contain only letters, digits, and underscores (_), and be no more than 128 characters long.
   VPC and vSwitch	If you do not have a VPC and vSwitch, DataWorks creates them automatically. If you have an existing VPC and vSwitch, select them. We recommend using the same VPC and vSwitch as the cloud services or storage you need to access.
   Service-linked role	Follow the on-screen instructions to click Create Service-linked Role.

3. Click Confirm Order and Pay.

If you select the DataWorks Basic Edition and a pay-as-you-go resource group, no payment is required in this step.

Step 3: Create a DataWorks workspace

A workspace is the basic management unit in DataWorks, used to isolate code, resources, and team members.

1. Log on to the DataWorks console and select a region in the upper-left corner.

2. On the Workspaces page, click Create Workspace.

3. Configure the basic information.

   Parameter	Description
   Workspace name	The name must start with a letter and can contain only letters, digits, and underscores (_).
   Display name	A user-friendly, customizable name.
   Isolate production and development environments	Specifies whether to enable Standard Mode. For single-user development, you can disable this option (Simple Mode).
   Workspace template	A predefined set of features. We recommend selecting All Capabilities.

4. Configure the advanced information.

   Parameter	Description
   Workspace administrator	If you are developing with a RAM user, add the RAM user as a workspace administrator.
   Create PAI workspace with the same name	Specifies whether to simultaneously create a PAI workspace. We recommend disabling this option.
   Default resource group	Used for scheduling tasks. Select the general-purpose resource group that you created in Step 2.
   Alibaba Cloud resource group	An identifier for classifying Alibaba Cloud resources. This parameter is optional.

5. Click Create Workspace.

Step 4: Associate a MaxCompute compute engine

Associate your MaxCompute project with the DataWorks workspace to use it as the compute engine.

1. Log on to the DataWorks console and select a region in the upper-left corner.

2. On the Workspaces page, click Manage in the Actions column for the target workspace.

3. On the Workspace Details page, click Computing Resource in the left navigation pane.

   On the Computing Resource page, click Associate Computing Resource, and select MaxCompute.

   Configure the following parameters:

   Parameter	Description
   MaxCompute project	Select the MaxCompute project that you created in Step 1.
   Default access identity	The identity used to execute jobs. We recommend selecting your Alibaba Cloud account.
   Endpoint	The service address. This parameter is automatically configured.
   Compute engine instance name	Enter a custom name.
   Resource group name	This enables communication between the compute engine and DataWorks. Select the general-purpose DataWorks resource group that you created in Step 2, and then click Test Connectivity to confirm that the connection is successful.

4. Click Confirm.

Step 5: Create a personal development environment

DataWorks notebook nodes must run on a personal development environment instance. Otherwise, you cannot execute code.

1. On the Workspaces page, in the Actions column for the target workspace, choose Shortcuts > Data Studio.

2. In the top menu, select Personal development environment, and click New Instance.

3. Configure the following parameters:

   Parameter	Description
   Instance name	Enter a custom instance name.
   Instance owner	Your Alibaba Cloud account or RAM user.
   Resource group	The group that contains the compute resources. Select the general-purpose resource group that you created.
   Resource type	Select CPU or GPU. For most scenarios, CPU is sufficient.
   Resource quota	Select an instance specification, such as ecs.g8i.xlarge (4 vCPUs, 16 GiB), which supports a 4-CU deduction.
   Select image	The runtime environment. Select an image version that includes MaxFrame, such as dataworks-maxcompute:py3.11-ubuntu20.04:py3.11-ubuntu20.04-202504-1.
   VPC	If you need to access services within a VPC, select the same VPC as your MaxCompute or OSS resources.
   vSwitch	A subnet. Select a vSwitch in the target availability zone under your VPC.
   Security group	A network access control mechanism. Select a configured security group that allows outbound traffic.
   Instance RAM role	The authorization role. We recommend selecting "DataWorks Default Role".

4. Click OK.

5. Wait 2 to 3 minutes for the instance status to change to Running.

Step 6: Create a notebook and run an example

1. On the Workspaces page, in the Actions column for the target workspace, choose Shortcuts > Data Studio.

2. From the top menu, select the personal development environment instance that you started.

3. On the Data Studio page, click the icon in the left navigation bar to go to the Data Studio page.

4. Right-click the target directory and choose Create Node > Notebook.

5. In the notebook editor, verify that the kernel in the upper-right corner is Python 3. Then, run the following code to check the MaxFrame version:

   import maxframe
   print("MaxFrame version:", maxframe.__version__)
   # If the version is earlier than 2.2.0, run the following command to upgrade:
   # !pip install maxframe -U

6. In a Python cell in your DataWorks notebook, you can use the built-in Magic Command to establish a connection to MaxCompute compute resources. This eliminates the need to repeatedly define connection information and expose plaintext AccessKeys in your code.

   import maxframe.dataframe as md
   import numpy as np
   from maxframe import new_session
   from maxframe.config import options
   from maxframe.udf import with_running_options
   from odps import ODPS
   import pandas as pd
   
   
   # -------------------------------
   # Initialize the MaxFrame Session
   # -------------------------------
   mf_session = %maxframe
   
   print("LogView address:", mf_session.get_logview_address())
   
   
   d = {'col1': [1, 2], 'col2': [3, 4]}
   df = md.DataFrame(data=d)
   print(df.execute().fetch())
   
   df = md.DataFrame(data=d, dtype=np.int8)
   print(df.execute().fetch())
   
   d = {'col1': [0, 1, 2, 3], 'col2': pd.Series([2, 3], index=[2, 3])}
   df = md.DataFrame(data=d, index=[0, 1, 2, 3])
   print(df.execute().fetch())
   
   data = np.array([(1, 2, 3), (4, 5, 6), (7, 8, 9)],
                   dtype=[("a", "i4"), ("b", "i4"), ("c", "i4")])
   df = md.DataFrame(data, columns=['c', 'a'])
   df.execute().fetch()
   
   mf_session.destroy()

Additional practices

from maxframe import options

# Use a pay-as-you-go quota to avoid running out of subscription resources.
options.session.quota_name = "Default pay-as-you-go quota"

# Control write concurrency to prevent overwhelming OSS.
options.sql.settings = {
    "odps.sql.split.dop": {"target_table": 5}
}

FAQ