Use MaxFrame in a DataWorks notebook

Prerequisites

Step 1: Activate MaxCompute

MaxCompute is an Alibaba Cloud data warehousing and computing service that supports the MaxFrame distributed Python computing framework.

1. Log in to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Manage Configurations > Projects.

3. On the Projects page, click Create Project. In the Create Project dialog box, configure the settings and click OK.

   • Region: Must be the same as the region you will select for DataWorks, for example, China (Shanghai).

   • Project Name (Globally Unique): Enter a custom name. We recommend including a business identifier.

   • Billing Method:

     • Subscription: Suitable for long-term, stable high-load production workloads. You pre-purchase a fixed number of compute units (CUs) to guarantee resource availability and avoid resource contention.

     • Pay-as-you-go: (Recommended) Suitable for short-term or testing phases. This method offers flexible billing based on the actual CUs consumed.

Step 2: Activate DataWorks

DataWorks is a big data development and governance platform that supports notebook development, job scheduling, data integration, and data governance.

1. Go to the DataWorks purchase page to activate a DataWorks edition.

   Parameter	Description	Recommended value
   DataWorks edition	Features and pricing vary by edition.	Basic Edition (Free, meets requirements for notebook development).
   Region and availability zone	Must be the same as the MaxCompute project region.	Select based on your business needs.

2. Go to the resource group purchase page to purchase compute resources for running tasks. Configure the following parameters:

   Parameter	Description
   Region and availability zone	Must be the same as the MaxCompute project region. Select based on your business needs.
   Billing method	pay-as-you-go (Recommended) subscription
   Resource group name	Enter a custom name. The name must start with a letter, contain only letters, digits, and underscores (_), and be no more than 128 characters long.
   VPC and vSwitch	If you do not have a VPC and vSwitch, DataWorks creates them automatically. If you have an existing VPC and vSwitch, select them. We recommend using the same VPC and vSwitch as the cloud services or storage you need to access.
   Service-linked role	Follow the on-screen instructions to click Create Service-linked Role.

3. Click Confirm Order and Pay.

If you select the DataWorks Basic Edition and a pay-as-you-go resource group, no payment is required in this step.

Step 3: Create a DataWorks workspace

A workspace is the basic management unit in DataWorks, used to isolate code, resources, and team members.

1. Log in to the DataWorks console and select a region in the upper-left corner.

2. On the Workspaces page, click Create Workspace.

3. Configure the basic information.

   Parameter	Description
   Workspace name	The name must start with a letter and can contain only letters, digits, and underscores (_).
   Display name	A user-friendly, customizable name.
   Isolate production and development environments	Specifies whether to enable Standard Mode. For single-user development, you can disable this option (Simple Mode).
   Workspace template	A predefined set of features. We recommend selecting All Capabilities.

4. Configure the advanced information.

   Parameter	Description
   Workspace administrator	If you are developing with a RAM user, add the RAM user as a workspace administrator.
   Create PAI workspace with the same name	Specifies whether to simultaneously create a PAI workspace. We recommend disabling this option.
   Default resource group	Used for scheduling tasks. Select the general-purpose resource group that you created in Step 2.
   Alibaba Cloud resource group	An identifier for classifying Alibaba Cloud resources. This parameter is optional.

5. Click Create Workspace.

Step 4: Associate a MaxCompute compute engine

Associate your MaxCompute project with the DataWorks workspace to use it as the compute engine.

1. Log in to the DataWorks console and select a region in the upper-left corner.

2. On the Workspaces page, in the Actions column for the target workspace, click Manage.

3. On the Workspace Details page, click Computing Resource in the navigation pane on the left.

   On the Computing Resource page, click Associate Computing Resources, and select MaxCompute.

   Configure the following parameters:

   Parameter	Description
   MaxCompute project	Select the MaxCompute project that you created in Step 1.
   Default access identity	The identity used to execute jobs. We recommend selecting your Alibaba Cloud account.
   Endpoint	The service address. This parameter is automatically configured.
   Compute engine instance name	Enter a custom name.
   Resource group name	This enables communication between the compute engine and DataWorks. Select the general-purpose DataWorks resource group that you created in Step 2, and then click Test Connectivity to confirm that the connection is successful.

4. Click Confirm.

Step 5: Create a personal development environment

DataWorks notebook nodes require a personal development environment instance to execute code.

1. On the Workspaces page, in the Actions column of the target workspace, choose Shortcuts > DataStudio.

2. In the top menu, select Personal Development Environment, and click New Instance.

3. Configure the following parameters:

   Parameter	Description
   Instance name	Enter a custom instance name.
   Instance owner	Your Alibaba Cloud account or RAM user.
   Resource group	The group that contains the compute resources. Select the general-purpose resource group that you created.
   Resource type	Select CPU or GPU. For most scenarios, CPU is sufficient.
   Resource quota	Select an instance specification, such as ecs.g8i.xlarge (4 vCPUs, 16 GiB), which supports a 4-CU deduction.
   Select image	The runtime environment. Select an image version that includes MaxFrame, such as dataworks-maxcompute:py3.11-ubuntu20.04:py3.11-ubuntu20.04-202504-1.
   VPC	If you need to access services within a VPC, select the same VPC as your MaxCompute or OSS resources.
   vSwitch	A subnet. Select a vSwitch in the target availability zone under your VPC.
   Security group	A network access control mechanism. Select a configured security group that allows outbound traffic.
   Instance RAM role	The authorization role. We recommend selecting "DataWorks Default Role".

4. Click OK.

5. Wait 2 to 3 minutes for the instance status to change to Running.

Step 6: Create a notebook and run an example

1. On the Workspaces page, in the Actions column of the target workspace, choose Shortcuts > DataStudio.

2. From the top menu, select the personal development environment instance that you started.

3. On the DataStudio page, click the icon in the left navigation bar to go to the Data Development page.

4. Right-click the target directory and choose Create Node > Notebook.

5. In the notebook editor, verify that the kernel in the upper-right corner is Python 3. Then, run the following code to check the MaxFrame version:

   import maxframe
   print("MaxFrame version:", maxframe.__version__)
   # If the version is earlier than 2.2.0, run the following command to upgrade:
   # !pip install maxframe -U

6. Use the built-in Magic Command in a Python cell to connect to MaxCompute compute resources without defining connection information or exposing plaintext AccessKeys in your code.

   import maxframe.dataframe as md
   import numpy as np
   from maxframe import new_session
   from maxframe.config import options
   from maxframe.udf import with_running_options
   from odps import ODPS
   import pandas as pd
   
   
   # -------------------------------
   # Initialize the MaxFrame Session
   # -------------------------------
   mf_session = %maxframe
   
   print("LogView address:", mf_session.get_logview_address())
   
   
   d = {'col1': [1, 2], 'col2': [3, 4]}
   df = md.DataFrame(data=d)
   print(df.execute().fetch())
   
   df = md.DataFrame(data=d, dtype=np.int8)
   print(df.execute().fetch())
   
   d = {'col1': [0, 1, 2, 3], 'col2': pd.Series([2, 3], index=[2, 3])}
   df = md.DataFrame(data=d, index=[0, 1, 2, 3])
   print(df.execute().fetch())
   
   data = np.array([(1, 2, 3), (4, 5, 6), (7, 8, 9)],
                   dtype=[("a", "i4"), ("b", "i4"), ("c", "i4")])
   df = md.DataFrame(data, columns=['c', 'a'])
   df.execute().fetch()
   
   mf_session.destroy()

Additional practices

from maxframe import options

# Use a pay-as-you-go quota to avoid running out of subscription resources.
options.session.quota_name = "Default pay-as-you-go quota"

# Control write concurrency to prevent overwhelming OSS.
options.sql.settings = {
    "odps.sql.split.dop": {"target_table": 5}
}

FAQ