DataWorks Notebook

Updated at:

This guide describes how to configure PySpark in DataWorks Notebook. Topics include custom configuration, OSS storage access, third-party Python packages, JAR resources, and Livy parameters.

Important

Upgrading dataworks-magic is a mandatory step to use the features described in this topic. To use all the configuration and Python packaging features, upgrade dataworks-magic to 0.1.19.0 or a later version and restart the Notebook kernel. On older versions, configurations may not be recognized or take effect, even if you can run %%maxcompute_spark.

Version requirements

The following table lists the minimum version required for each feature.

Feature

Minimum dataworks-magic version

Description

spark_conf

0.1.15.2

Earlier versions do not support the Spark configuration method described in this guide.

livy_server_conf

0.1.19.0

Older versions use the functionally equivalent config. As of version 0.1.19.0, the new name livy_server_conf is recommended for its clearer purpose.

maxcompute_spark package

0.1.19.0

Earlier versions do not support the command for packaging third-party Python packages within the notebook.

Therefore, to use all the features in this document, you must meet the following requirement: dataworks-magic >= 0.1.19.0.

You can check the current version in a notebook Python cell:

import dataworks
print(dataworks.__version__)

Custom configurations

Before you start, ensure that your dataworks-magic version meets the requirements. If an upgrade is necessary, perform it in your DataWorks development environment and then restart the notebook kernel to load the new version.

Cell configuration is divided into the following sections by purpose:

Configuration section

Purpose

livy_server_conf

Configures Livy Server specifications, lifecycle, and image.

spark_conf

Configures Spark driver, executor, and jobs.

client_conf

Configures local dataworks-magic client behavior, such as waiting, retries, and logging.

quota

Specifies the MaxCompute quota for Livy to use.

  • livy_server_conf

    This is used to configure the specifications, lifecycle, and image of the Livy Server. It is functionally equivalent to the config parameter used in older versions. The only difference is the parameter name, which does not affect the meaning of the Livy parameters themselves. Version 0.1.19.0 introduced the new name livy_server_conf to make its role clearer and easier to understand, and to distinguish it from the Spark parameter spark_conf and the client parameter client_conf.

    For backward compatibility with existing Notebooks, the current version still accepts the old name config, but we recommend using livy_server_conf. If both are specified, livy_server_conf takes precedence, and config is ignored.

    The following example sets the Livy specifications and maximum idle time, and enables the three-layer model for Spark:

    %%maxcompute_spark
    {
        "livy_server_conf": {
            "cpu": 1,
            "memory": "4096M",
            "livy.server.max.idle.time": "10m"
        },
        "quota": "<your_quota>",
        "spark_conf": {
            "spark.sql.catalog.odps.enableNamespaceSchema": "true"
        }
    }

    Typically, you do not need to manually specify the Livy name. By default, maxcompute_spark enables LivyPool, which reuses suitable Livy and Spark sessions based on conditions such as the project, quota, Livy configuration, Spark configuration, and language. If no reusable sessions are available, new ones are automatically created.

    If you rerun the configuration and it does not match the cached session, the system requests or reuses a different session. Switching sessions discards the previous session's runtime state, such as variables and temporary views.

  • Common parameters

    • --force: Skips cache matching for existing sessions and forces the creation of a new session.

    • --no-use-pool: Disables LivyPool and uses the legacy Livy management process. This option is not recommended.

    • client_conf: Controls only the behavior of the Notebook client and is not submitted as a Livy or Spark configuration. For example:

      %%maxcompute_spark
      {
          "client_conf": {
              "wait_for_running_timeout_seconds": 900,
              "livy_session_startup_timeout_seconds": 900,
              "retry_seconds_to_sleep_list": [5, 10, 15]
          },
          "spark_conf": {
              "spark.executor.instances": "4"
          }
      }

OSS storage

Configure the OSS access parameters:

Important
  • oss_endpoint must use an Internal endpoint, for example: oss-cn-shanghai-internal.aliyuncs.com.

  • Do not set the spark.hadoop.fs.oss.impl and spark.hadoop.fs.AbstractFileSystem.oss.impl parameters in a Notebook, or you may be unable to access OSS.

%%maxcompute_spark
{
  "spark_conf": {
    "spark.hadoop.odps.cupid.trusted.services.access.list": "<bucket-name>.<oss-endpoint>",
    "spark.hadoop.fs.oss.accessKeyId": "***",
    "spark.hadoop.fs.oss.accessKeySecret": "***",
    "spark.hadoop.fs.oss.endpoint": "<oss-endpoint>"
  }
}

Third-party Python packages

Packaging with maxcompute_spark package (recommended)

The package command calls pyodps-pack on the server side to package dependencies into a .tar.gz file and directly upload the file as a MaxCompute Archive Resource. You no longer need to install pyodps, run pyodps-pack, or use odpscmd to upload packages locally in a Notebook.

Compared to local pyodps-pack, packaging with DataWorks Notebook is more lightweight and does not require you to prepare or run Docker. Due to limitations in the current Spark Python environment, it currently supports Python 3.6, 3.7, and 3.11.

Note

We recommend against using pyodps-pack locally. While pyodps-pack can generate a dependency package for other Python versions, this does not guarantee that the package will run successfully on a MaxCompute Spark cluster. If your local Python version, operating system, glibc, CPU architecture, or underlying dynamic libraries are inconsistent with the Spark cluster's environment, you may encounter runtime errors, especially when dependencies contain .so files. These errors can include import failures, missing dynamic libraries, or binary incompatibilities. Unless you have explicitly replicated the Spark cluster environment, we recommend using the versions supported by DataWorks Notebook (3.6, 3.7, or 3.11) for packaging.

We recommend that you use Python 3.11. Versions 3.6, 3.7, and 3.11 are supported, and the command automatically selects the corresponding Livy image for the packaging task. Python 3.11 uses the Alinux3 image.

pack_conf.pythonVersion must match the Python version that is used to run the Spark Session. The package command only determines the Python version used to pack dependencies and does not automatically switch the Python environment for the subsequent Spark Session. If the versions do not match, runtime errors such as import failures or dynamic library incompatibility may occur in the Spark Driver or Executor, even if the packaging is successful.

By default, Spark in the notebook uses Python 3.11. The version mappings are as follows:

Packaging pythonVersion

Final Spark Python version

Required Spark session configuration

3.11

Default Python 3.11

No Python environment switch is needed.

3.6

Python 3.6.8

Set spark.hadoop.odps.spark.alinux3.enabled=false.

Non-Alinux3 environments use Python 3.6.8 by default, so no other Python environment parameters need to be set.

3.7

Python 3.7.9

Set the following parameters:

  • spark.hadoop.odps.spark.alinux3.enabled=false

  • spark.hadoop.odps.cupid.resources=public.python-3.7.9-ucs4.tar.gz

  • spark.pyspark.python=./public.python-3.7.9-ucs4.tar.gz/python-3.7.9-ucs4/bin/python3

  • When using Python 3.6.8, you need at least the following configuration to start the final Spark session:

    %%maxcompute_spark
    {
        "spark_conf": {
            "spark.hadoop.odps.spark.alinux3.enabled": "false"
        }
    }
  • When using Python 3.7.9, you need the following configuration to start the final Spark session:

    %%maxcompute_spark
    {
        "spark_conf": {
            "spark.hadoop.odps.spark.alinux3.enabled": "false",
            "spark.hadoop.odps.cupid.resources": "public.python-3.7.9-ucs4.tar.gz",
            "spark.pyspark.python": "./public.python-3.7.9-ucs4.tar.gz/python-3.7.9-ucs4/bin/python3"
        }
    }

For more information about Python environment parameters, see PySpark Python versions and dependencies. When you load the packaged dependency resource, you also need to configure cupid.resources and PYTHONPATH as described in Use the packaging result in a Spark session.

The simplest usage is as follows. Multiple dependencies must be separated by newlines, not commas or spaces:

%%maxcompute_spark package
{
    "pack_conf": {
        "pythonVersion": "3.11",
        "requirements": "requests==2.32.3\nnumpy==1.26.4\npandas==2.2.3\npyarrow==17.0.0",
        "packName": "py311-deps.tar.gz"
    }
}

Alternatively, you can write the requirements as multi-line text and indent subsequent lines to align with the content of the first line. dataworks-magic automatically handles line breaks and removes the leading and trailing whitespace from each line. Therefore, these spaces used for formatting will not be included in the final requirements content:

%%maxcompute_spark package
{
    "pack_conf": {
        "pythonVersion": "3.11",
        "requirements": "requests==2.32.3\n                     numpy==1.26.4\n                     pandas==2.2.3\n                     pyarrow==17.0.0",
        "packName": "py311-deps.tar.gz"
    }
}

The pack_conf parameter is described as follows:

Parameter

Required

Description

pythonVersion

Yes

Supports 3.6, 3.7, and 3.11. The corresponding full version numbers are also supported.

requirements

At least one dependency source must be provided.

A multi-line string in requirements.txt format.

sources

At least one dependency source must be provided.

Uses an existing requirements resource or Python project. Pass a single object for a single source, or an array for multiple sources.

packProject

No

Specifies the MaxCompute project to which the packaged artifact is uploaded. If omitted, it defaults to the project associated with the current DataWorks computing resources.

packName

No

The name of the output Archive Resource. If this parameter is left empty, a name is automatically generated and printed in the log. An explicitly specified name must end with .tar.gz .

overwrite

No

Specifies whether to overwrite a resource with the same name. Overwriting is enabled by default and usually does not need to be specified.

timeout

No

The default server-side packaging timeout is 1h. Set this value explicitly only when the estimated time exceeds 1 hour, for example, 2h.

  • packProject determines the MaxCompute Project to which the packaged artifact is uploaded. If you do not specify this parameter, the MaxCompute Project that is associated with the current DataWorks compute resource is used by default.

  • If you do not specify a value for packName, a name is automatically generated and printed in the packaging log. We recommend that you explicitly specify a name that ends with .tar.gz to make it easier to reference the packaged artifact and to use the same name for repackaging.

  • The timeout parameter is set to 1h by default. You need to explicitly configure this parameter only if the estimated packaging time exceeds 1 hour, for example, "timeout": "2h".

  • Do not set the packaging timeout in client_conf.package_timeout_seconds because this field is ignored. Use pack_conf.timeout instead. The package command also does not use spark_conf.

For most scenarios, use a single dependency source:

  • Directly specify third-party packages using requirements

  • Use a single sources object to reference existing files or projects.

  • Advanced combination scenarios require you to specify both requirements and sources, or provide an array in sources.

All dependencies are ultimately merged into a single output package.

Packaging from an existing resource

Most tasks require only one dependency source. For a single source, you can set sources to an object directly instead of wrapping it in an array.

The following table shows the recommended approach for common scenarios.

Scenario

Recommended approach

Directly specify PyPI third-party packages

Use Section 3.1's pack_conf.requirements

A requirements.txt file has already been uploaded to MaxCompute

Use sources.type: "requirements"

You need to install your own Python project

Use sources.type: "project"

Note the difference between two similarly named fields:

  • pack_conf.requirements

    Enter the requirements directly, for example, pandas==2.2.3.

  • sources.type: "requirements"

    resource specifies the name of the Resource for an existing requirements file in MaxCompute.

Requirements resource

Suppose a MaxCompute project contains a FILE Resource named requirements.txt. You can package it as follows:

%%maxcompute_spark package
{
    "pack_conf": {
        "pythonVersion": "3.11",
        "sources": {
            "type": "requirements",
            "resource": "<your_project>.requirements.txt"
        },
        "packName": "py311-requirements.tar.gz"
    }
}

type: "requirements" indicates that an existing requirements file is read.

The resource must be in the <project>.<resourceName> format. The file content must follow the requirements.txt syntax, and each dependency must be on a separate line.

Python project resource

If you need to package your Python project with its dependencies, first upload the project as a MaxCompute FILE or ARCHIVE Resource, and then use type: "project":

%%maxcompute_spark package
{
    "pack_conf": {
        "pythonVersion": "3.11",
        "sources": {
            "type": "project",
            "resource": "<your_project>.my_python_project.tar.gz"
        },
        "packName": "py311-project.tar.gz"
    }
}

The project resource must be a .zip or .tar.gz file. The extracted project directory must contain setup.py or pyproject.toml. If the actual project is located in a subdirectory within the archive, specify the subdir parameter:

"sources": {
    "type": "project",
    "resource": "<your_project>.my_python_project.tar.gz",
    "subdir": "python"
}

Packaging from multiple sources

Define sources as an array only when you need to merge multiple requirements files or Python projects. All sources are processed in a single task, and ultimately, only one Archive Resource is generated for the specified packName.

%%maxcompute_spark package
{
    "pack_conf": {
        "pythonVersion": "3.11",
        "sources": [
            {
                "type": "requirements",
                "resource": "<your_project>.requirements.txt"
            },
            {
                "type": "project",
                "resource": "<your_project>.my_python_project.tar.gz"
            }
        ],
        "packName": "py311-all-deps.tar.gz"
    }
}

Version constraints and Python package names from multiple sources must not conflict. When you directly enter dependency text, the top-level requirements field from section 3.1 is generally preferred. type: "inline" is still compatible but is usually unnecessary. The full Livy types, requirements_text, requirements_resource, and project_resource, also remain compatible.

Livy instances for packaging tasks

By default, a package prioritizes reusing a running Livy instance in the same Project with a matching image type. Python 3.11 matches only Alinux3 Livy, and Python 3.6/3.7 matches only non-Alinux3 Livy.

If no reusable Livy is available, the command creates a pydeps-* Livy. By default, the newly created or reused Livy is not automatically deleted so that it can be reused by subsequent tasks.

To force the creation of a temporary, single-use Livy that is cleaned up after the task completes or fails, use the --force flag:

%%maxcompute_spark package --force
{
    "pack_conf": {
        "pythonVersion": "3.11",
        "requirements": "requests==2.32.3\nnumpy==1.26.4",
        "packName": "py311-deps.tar.gz"
    }
}

The packaging process outputs the Livy Logview URL, task ID, progress, and target resource. For detailed logs, check the driver stderr in the corresponding Livy Logview.

Using packaged results in a Spark session

After the packaging is successful, configure the generated Archive Resource in Spark. The following is a sample configuration for the default Python 3.11 in a Notebook. Multiple cupid.resources values are separated by commas, and multiple PYTHONPATH values are separated by colons.

Because the dependencies for pyodps-pack are in the packages subdirectory of the archive, configure the settings as follows:

%%maxcompute_spark
{
    "spark_conf": {
        "spark.hadoop.odps.cupid.resources": "<your_project>.py311-deps.tar.gz",
        "spark.executorEnv.PYTHONPATH": "./<your_project>.py311-deps.tar.gz/packages",
        "spark.yarn.appMasterEnv.PYTHONPATH": "./<your_project>.py311-deps.tar.gz/packages"
    }
}

If you use pythonVersion: "3.6" for packaging, you must also add the following to the spark_conf mentioned above:

"spark.hadoop.odps.spark.alinux3.enabled": "false"

Non-Alinux3 environments use Python 3.6.8 by default. You do not need to set any other Python environment parameters except for cupid.resources and PYTHONPATH, which are required to load packaged artifacts.

If you use pythonVersion: "3.7" during packaging, you must load both the public Python 3.7.9 environment and the dependencies generated in this process. A complete example is as follows:

%%maxcompute_spark
{
    "spark_conf": {
        "spark.hadoop.odps.spark.alinux3.enabled": "false",
        "spark.hadoop.odps.cupid.resources": "public.python-3.7.9-ucs4.tar.gz,<your_project>.py37-deps.tar.gz",
        "spark.pyspark.python": "./public.python-3.7.9-ucs4.tar.gz/python-3.7.9-ucs4/bin/python3",
        "spark.executorEnv.PYTHONPATH": "./<your_project>.py37-deps.tar.gz/packages",
        "spark.yarn.appMasterEnv.PYTHONPATH": "./<your_project>.py37-deps.tar.gz/packages"
    }
}

You cannot keep only one of these cupid.resources files: public.python-3.7.9-ucs4.tar.gz provides the Python 3.7.9 runtime environment, and <your_project>.py37-deps.tar.gz provides the third-party dependencies included in this package.

For details on cupid.resources, see Data Interoperability Configuration. For details on PYTHONPATH, see Using Custom Python Packages. If the archive is not generated by pyodps-pack, set PYTHONPATH according to the actual directory structure.

Manual packaging

Consider manual execution only when you cannot use maxcompute_spark package and can ensure that the local packaging environment is consistent with the target Spark cluster environment:

pip install pyodps
pyodps-pack -r requirements.txt --python-version=3.11 -o py311-deps.tar.gz

Then, upload the package by using odpscmd:

add archive /path/to/py311-deps.tar.gz -f;

JAR or other resource packages

  1. Upload the resource. For example, to upload a .tar.gz archive: add archive /path/to/<package-name> -f;

  2. Configure Spark to load the resource. Separate multiple resources with commas (,).

    %%maxcompute_spark
    {
      "spark_conf": {
        "spark.hadoop.odps.cupid.resources": "<your_project>.<package-name>"
      }
    }

This can be used in combination with the Python package configuration, and both are specified in the spark_conf parameter.

Advanced configuration

Specify Spark version

  • Supported versions are as follows:

    • spark-3.1.1-odps0.47.0

    • spark-3.4.2-odps0.48.0 (default)

    • spark-3.5.2-odps0.49.0

  • The available versions depend on your current MaxCompute environment. To set the version:

    %%maxcompute_spark
    {
      "spark_conf": {
        "spark.hadoop.odps.spark.version": "spark-3.4.2-odps0.48.0"
      }
    }

Select a Spark Python environment

By default, Spark in the notebook uses Python 3.11.

  • When you switch to Python 3.6.8 or Python 3.7.9, you need to set spark.hadoop.odps.spark.alinux3.enabled=false in spark_conf.

  • Python 3.7.9 also requires you to load public.python-3.7.9-ucs4.tar.gz and set spark.pyspark.python. For the complete configuration, see Sections 3.1 and 3.4.

livy_server_conf manages Livy Server parameters and cannot replace spark_conf, which is required to switch the Spark Python environment. The pack_conf.pythonVersion used for packaging, the Python version used by the Spark session, and the binary environment of the dependency packages must be consistent.

For more information, see PySpark Python versions and dependencies.

Plot with Matplotlib

First, package and upload Matplotlib as described in Using third-party Python packages. Then, create plots in the notebook:

%%spark

import matplotlib.pyplot as plt
import io, base64, json
import matplotlib
# Optional
matplotlib.use('Agg')

# Crucial step: Create the figure here first.
fig = plt.figure()

# Example plot
x = [1, 2, 3, 4]
y = [20, 22, 19, 23]

plt.plot(x, y, marker='o', linestyle='--', color='b')
plt.title("Temperature Change Over Time")
plt.xlabel("Time")
plt.ylabel("Temperature")

# Render the image in the notebook (Livy magic command)
%matplot plt

Livy configuration

You can refer to the Apache Livy Configuration, but we recommend against modifying most parameters. The following table lists commonly used configurations.

Parameter

Default

Description

livy.server.access-control.enabled

true

Specifies whether to enable access control. It is enabled by default, and disabling it is not recommended.

livy.server.max.alive.time

3d

The maximum lifetime of the Livy Server. The default is 3 days. Set the value to 0 to disable this limit.

livy.server.max.alive.session.wait.time

1d

After the Livy Server reaches its maximum lifecycle, this is the maximum time it waits for existing sessions and packaging tasks to finish. No new tasks are accepted during this period.

livy.server.max.idle.time

1h

The maximum time a Livy Server can remain idle without any sessions.

livy.server.session.timeout

1h

The timeout period for an inactive Spark session.

livy.server.session.timeout-check.skip-busy

false

When true, busy sessions that still have tasks running are not subject to the idle timeout.

livy.server.session.state-retain.sec

300s

The duration for which the state of a finished session is retained in Livy's memory for querying. This does not affect session restart.

livy.server.session.max-creation

50

The maximum number of sessions that can exist simultaneously on a single Livy Server. Default: 50.

Time parameters usually support s, m, h, and d, such as 30m, 12h, or 3d.