DataWorks Notebook
This guide describes how to configure PySpark in DataWorks Notebook. Topics include custom configuration, OSS storage access, third-party Python packages, JAR resources, and Livy parameters.
Upgrading dataworks-magic is a mandatory step to use the features described in this topic. To use all the configuration and Python packaging features, upgrade dataworks-magic to 0.1.19.0 or a later version and restart the Notebook kernel. On older versions, configurations may not be recognized or take effect, even if you can run %%maxcompute_spark.
Version requirements
The following table lists the minimum version required for each feature.
Feature | Minimum dataworks-magic version | Description |
|
| Earlier versions do not support the Spark configuration method described in this guide. |
|
| Older versions use the functionally equivalent |
|
| Earlier versions do not support the command for packaging third-party Python packages within the notebook. |
Therefore, to use all the features in this document, you must meet the following requirement: dataworks-magic >= 0.1.19.0.
You can check the current version in a notebook Python cell:
import dataworks
print(dataworks.__version__)Custom configurations
Before you start, ensure that your dataworks-magic version meets the requirements. If an upgrade is necessary, perform it in your DataWorks development environment and then restart the notebook kernel to load the new version.
Cell configuration is divided into the following sections by purpose:
Configuration section | Purpose |
| Configures Livy Server specifications, lifecycle, and image. |
| Configures Spark driver, executor, and jobs. |
| Configures local |
| Specifies the MaxCompute quota for Livy to use. |
livy_server_confThis is used to configure the specifications, lifecycle, and image of the Livy Server. It is functionally equivalent to the
configparameter used in older versions. The only difference is the parameter name, which does not affect the meaning of the Livy parameters themselves. Version0.1.19.0introduced the new namelivy_server_confto make its role clearer and easier to understand, and to distinguish it from the Spark parameterspark_confand the client parameterclient_conf.For backward compatibility with existing Notebooks, the current version still accepts the old name
config, but we recommend usinglivy_server_conf. If both are specified,livy_server_conftakes precedence, andconfigis ignored.The following example sets the Livy specifications and maximum idle time, and enables the three-layer model for Spark:
%%maxcompute_spark { "livy_server_conf": { "cpu": 1, "memory": "4096M", "livy.server.max.idle.time": "10m" }, "quota": "<your_quota>", "spark_conf": { "spark.sql.catalog.odps.enableNamespaceSchema": "true" } }Typically, you do not need to manually specify the Livy
name. By default,maxcompute_sparkenables LivyPool, which reuses suitable Livy and Spark sessions based on conditions such as the project, quota, Livy configuration, Spark configuration, and language. If no reusable sessions are available, new ones are automatically created.If you rerun the configuration and it does not match the cached session, the system requests or reuses a different session. Switching sessions discards the previous session's runtime state, such as variables and temporary views.
Common parameters
--force: Skips cache matching for existing sessions and forces the creation of a new session.--no-use-pool: Disables LivyPool and uses the legacy Livy management process. This option is not recommended.client_conf: Controls only the behavior of the Notebook client and is not submitted as a Livy or Spark configuration. For example:%%maxcompute_spark { "client_conf": { "wait_for_running_timeout_seconds": 900, "livy_session_startup_timeout_seconds": 900, "retry_seconds_to_sleep_list": [5, 10, 15] }, "spark_conf": { "spark.executor.instances": "4" } }
OSS storage
Configure the OSS access parameters:
oss_endpointmust use an Internal endpoint, for example:oss-cn-shanghai-internal.aliyuncs.com.Do not set the
spark.hadoop.fs.oss.implandspark.hadoop.fs.AbstractFileSystem.oss.implparameters in a Notebook, or you may be unable to access OSS.
%%maxcompute_spark
{
"spark_conf": {
"spark.hadoop.odps.cupid.trusted.services.access.list": "<bucket-name>.<oss-endpoint>",
"spark.hadoop.fs.oss.accessKeyId": "***",
"spark.hadoop.fs.oss.accessKeySecret": "***",
"spark.hadoop.fs.oss.endpoint": "<oss-endpoint>"
}
}Third-party Python packages
Packaging with maxcompute_spark package (recommended)
The package command calls pyodps-pack on the server side to package dependencies into a .tar.gz file and directly upload the file as a MaxCompute Archive Resource. You no longer need to install pyodps, run pyodps-pack, or use odpscmd to upload packages locally in a Notebook.
Compared to local pyodps-pack, packaging with DataWorks Notebook is more lightweight and does not require you to prepare or run Docker. Due to limitations in the current Spark Python environment, it currently supports Python 3.6, 3.7, and 3.11.
We recommend against using pyodps-pack locally. While pyodps-pack can generate a dependency package for other Python versions, this does not guarantee that the package will run successfully on a MaxCompute Spark cluster. If your local Python version, operating system, glibc, CPU architecture, or underlying dynamic libraries are inconsistent with the Spark cluster's environment, you may encounter runtime errors, especially when dependencies contain .so files. These errors can include import failures, missing dynamic libraries, or binary incompatibilities. Unless you have explicitly replicated the Spark cluster environment, we recommend using the versions supported by DataWorks Notebook (3.6, 3.7, or 3.11) for packaging.
We recommend that you use Python 3.11. Versions 3.6, 3.7, and 3.11 are supported, and the command automatically selects the corresponding Livy image for the packaging task. Python 3.11 uses the Alinux3 image.
pack_conf.pythonVersion must match the Python version that is used to run the Spark Session. The package command only determines the Python version used to pack dependencies and does not automatically switch the Python environment for the subsequent Spark Session. If the versions do not match, runtime errors such as import failures or dynamic library incompatibility may occur in the Spark Driver or Executor, even if the packaging is successful.
By default, Spark in the notebook uses Python 3.11. The version mappings are as follows:
Packaging | Final Spark Python version | Required Spark session configuration |
| Default Python 3.11 | No Python environment switch is needed. |
| Python 3.6.8 | Set Non-Alinux3 environments use Python 3.6.8 by default, so no other Python environment parameters need to be set. |
| Python 3.7.9 | Set the following parameters:
|
When using Python 3.6.8, you need at least the following configuration to start the final Spark session:
%%maxcompute_spark { "spark_conf": { "spark.hadoop.odps.spark.alinux3.enabled": "false" } }When using Python 3.7.9, you need the following configuration to start the final Spark session:
%%maxcompute_spark { "spark_conf": { "spark.hadoop.odps.spark.alinux3.enabled": "false", "spark.hadoop.odps.cupid.resources": "public.python-3.7.9-ucs4.tar.gz", "spark.pyspark.python": "./public.python-3.7.9-ucs4.tar.gz/python-3.7.9-ucs4/bin/python3" } }
For more information about Python environment parameters, see PySpark Python versions and dependencies. When you load the packaged dependency resource, you also need to configure cupid.resources and PYTHONPATH as described in Use the packaging result in a Spark session.
The simplest usage is as follows. Multiple dependencies must be separated by newlines, not commas or spaces:
%%maxcompute_spark package
{
"pack_conf": {
"pythonVersion": "3.11",
"requirements": "requests==2.32.3\nnumpy==1.26.4\npandas==2.2.3\npyarrow==17.0.0",
"packName": "py311-deps.tar.gz"
}
}Alternatively, you can write the requirements as multi-line text and indent subsequent lines to align with the content of the first line. dataworks-magic automatically handles line breaks and removes the leading and trailing whitespace from each line. Therefore, these spaces used for formatting will not be included in the final requirements content:
%%maxcompute_spark package
{
"pack_conf": {
"pythonVersion": "3.11",
"requirements": "requests==2.32.3\n numpy==1.26.4\n pandas==2.2.3\n pyarrow==17.0.0",
"packName": "py311-deps.tar.gz"
}
}The pack_conf parameter is described as follows:
Parameter | Required | Description |
| Yes | Supports |
| At least one dependency source must be provided. | A multi-line string in requirements.txt format. |
| At least one dependency source must be provided. | Uses an existing requirements resource or Python project. Pass a single object for a single source, or an array for multiple sources. |
| No | Specifies the MaxCompute project to which the packaged artifact is uploaded. If omitted, it defaults to the project associated with the current DataWorks computing resources. |
| No | The name of the output Archive Resource. If this parameter is left empty, a name is automatically generated and printed in the log. An explicitly specified name must end with |
| No | Specifies whether to overwrite a resource with the same name. Overwriting is enabled by default and usually does not need to be specified. |
| No | The default server-side packaging timeout is |
packProjectdetermines the MaxCompute Project to which the packaged artifact is uploaded. If you do not specify this parameter, the MaxCompute Project that is associated with the current DataWorks compute resource is used by default.If you do not specify a value for
packName, a name is automatically generated and printed in the packaging log. We recommend that you explicitly specify a name that ends with.tar.gzto make it easier to reference the packaged artifact and to use the same name for repackaging.The
timeoutparameter is set to1hby default. You need to explicitly configure this parameter only if the estimated packaging time exceeds 1 hour, for example,"timeout": "2h".Do not set the packaging timeout in
client_conf.package_timeout_secondsbecause this field is ignored. Usepack_conf.timeoutinstead. Thepackagecommand also does not usespark_conf.
For most scenarios, use a single dependency source:
Directly specify third-party packages using
requirementsUse a single
sourcesobject to reference existing files or projects.Advanced combination scenarios require you to specify both
requirementsandsources, or provide an array insources.
All dependencies are ultimately merged into a single output package.
Packaging from an existing resource
Most tasks require only one dependency source. For a single source, you can set sources to an object directly instead of wrapping it in an array.
The following table shows the recommended approach for common scenarios.
Scenario | Recommended approach |
Directly specify PyPI third-party packages | Use Section 3.1's |
A requirements.txt file has already been uploaded to MaxCompute | Use |
You need to install your own Python project | Use |
Note the difference between two similarly named fields:
pack_conf.requirementsEnter the requirements directly, for example,
pandas==2.2.3.sources.type: "requirements"resourcespecifies the name of the Resource for an existing requirements file in MaxCompute.
Requirements resource
Suppose a MaxCompute project contains a FILE Resource named requirements.txt. You can package it as follows:
%%maxcompute_spark package
{
"pack_conf": {
"pythonVersion": "3.11",
"sources": {
"type": "requirements",
"resource": "<your_project>.requirements.txt"
},
"packName": "py311-requirements.tar.gz"
}
}type: "requirements" indicates that an existing requirements file is read.
The resource must be in the <project>.<resourceName> format. The file content must follow the requirements.txt syntax, and each dependency must be on a separate line.
Python project resource
If you need to package your Python project with its dependencies, first upload the project as a MaxCompute FILE or ARCHIVE Resource, and then use type: "project":
%%maxcompute_spark package
{
"pack_conf": {
"pythonVersion": "3.11",
"sources": {
"type": "project",
"resource": "<your_project>.my_python_project.tar.gz"
},
"packName": "py311-project.tar.gz"
}
}The project resource must be a .zip or .tar.gz file. The extracted project directory must contain setup.py or pyproject.toml. If the actual project is located in a subdirectory within the archive, specify the subdir parameter:
"sources": {
"type": "project",
"resource": "<your_project>.my_python_project.tar.gz",
"subdir": "python"
}Packaging from multiple sources
Define sources as an array only when you need to merge multiple requirements files or Python projects. All sources are processed in a single task, and ultimately, only one Archive Resource is generated for the specified packName.
%%maxcompute_spark package
{
"pack_conf": {
"pythonVersion": "3.11",
"sources": [
{
"type": "requirements",
"resource": "<your_project>.requirements.txt"
},
{
"type": "project",
"resource": "<your_project>.my_python_project.tar.gz"
}
],
"packName": "py311-all-deps.tar.gz"
}
}Version constraints and Python package names from multiple sources must not conflict. When you directly enter dependency text, the top-level requirements field from section 3.1 is generally preferred. type: "inline" is still compatible but is usually unnecessary. The full Livy types, requirements_text, requirements_resource, and project_resource, also remain compatible.
Livy instances for packaging tasks
By default, a package prioritizes reusing a running Livy instance in the same Project with a matching image type. Python 3.11 matches only Alinux3 Livy, and Python 3.6/3.7 matches only non-Alinux3 Livy.
If no reusable Livy is available, the command creates a pydeps-* Livy. By default, the newly created or reused Livy is not automatically deleted so that it can be reused by subsequent tasks.
To force the creation of a temporary, single-use Livy that is cleaned up after the task completes or fails, use the --force flag:
%%maxcompute_spark package --force
{
"pack_conf": {
"pythonVersion": "3.11",
"requirements": "requests==2.32.3\nnumpy==1.26.4",
"packName": "py311-deps.tar.gz"
}
}The packaging process outputs the Livy Logview URL, task ID, progress, and target resource. For detailed logs, check the driver stderr in the corresponding Livy Logview.
Using packaged results in a Spark session
After the packaging is successful, configure the generated Archive Resource in Spark. The following is a sample configuration for the default Python 3.11 in a Notebook. Multiple cupid.resources values are separated by commas, and multiple PYTHONPATH values are separated by colons.
Because the dependencies for pyodps-pack are in the packages subdirectory of the archive, configure the settings as follows:
%%maxcompute_spark
{
"spark_conf": {
"spark.hadoop.odps.cupid.resources": "<your_project>.py311-deps.tar.gz",
"spark.executorEnv.PYTHONPATH": "./<your_project>.py311-deps.tar.gz/packages",
"spark.yarn.appMasterEnv.PYTHONPATH": "./<your_project>.py311-deps.tar.gz/packages"
}
}If you use pythonVersion: "3.6" for packaging, you must also add the following to the spark_conf mentioned above:
"spark.hadoop.odps.spark.alinux3.enabled": "false"Non-Alinux3 environments use Python 3.6.8 by default. You do not need to set any other Python environment parameters except for cupid.resources and PYTHONPATH, which are required to load packaged artifacts.
If you use pythonVersion: "3.7" during packaging, you must load both the public Python 3.7.9 environment and the dependencies generated in this process. A complete example is as follows:
%%maxcompute_spark
{
"spark_conf": {
"spark.hadoop.odps.spark.alinux3.enabled": "false",
"spark.hadoop.odps.cupid.resources": "public.python-3.7.9-ucs4.tar.gz,<your_project>.py37-deps.tar.gz",
"spark.pyspark.python": "./public.python-3.7.9-ucs4.tar.gz/python-3.7.9-ucs4/bin/python3",
"spark.executorEnv.PYTHONPATH": "./<your_project>.py37-deps.tar.gz/packages",
"spark.yarn.appMasterEnv.PYTHONPATH": "./<your_project>.py37-deps.tar.gz/packages"
}
}You cannot keep only one of these cupid.resources files: public.python-3.7.9-ucs4.tar.gz provides the Python 3.7.9 runtime environment, and <your_project>.py37-deps.tar.gz provides the third-party dependencies included in this package.
For details on cupid.resources, see Data Interoperability Configuration. For details on PYTHONPATH, see Using Custom Python Packages. If the archive is not generated by pyodps-pack, set PYTHONPATH according to the actual directory structure.
Manual packaging
Consider manual execution only when you cannot use maxcompute_spark package and can ensure that the local packaging environment is consistent with the target Spark cluster environment:
pip install pyodps
pyodps-pack -r requirements.txt --python-version=3.11 -o py311-deps.tar.gzThen, upload the package by using odpscmd:
add archive /path/to/py311-deps.tar.gz -f;JAR or other resource packages
Upload the resource. For example, to upload a
.tar.gzarchive:add archive /path/to/<package-name> -f;Configure Spark to load the resource. Separate multiple resources with commas (,).
%%maxcompute_spark { "spark_conf": { "spark.hadoop.odps.cupid.resources": "<your_project>.<package-name>" } }
This can be used in combination with the Python package configuration, and both are specified in the spark_conf parameter.
Advanced configuration
Specify Spark version
Supported versions are as follows:
spark-3.1.1-odps0.47.0spark-3.4.2-odps0.48.0(default)spark-3.5.2-odps0.49.0
The available versions depend on your current MaxCompute environment. To set the version:
%%maxcompute_spark { "spark_conf": { "spark.hadoop.odps.spark.version": "spark-3.4.2-odps0.48.0" } }
Select a Spark Python environment
By default, Spark in the notebook uses Python 3.11.
When you switch to Python 3.6.8 or Python 3.7.9, you need to set
spark.hadoop.odps.spark.alinux3.enabled=falseinspark_conf.Python 3.7.9 also requires you to load
public.python-3.7.9-ucs4.tar.gzand setspark.pyspark.python. For the complete configuration, see Sections 3.1 and 3.4.
livy_server_conf manages Livy Server parameters and cannot replace spark_conf, which is required to switch the Spark Python environment. The pack_conf.pythonVersion used for packaging, the Python version used by the Spark session, and the binary environment of the dependency packages must be consistent.
For more information, see PySpark Python versions and dependencies.
Plot with Matplotlib
First, package and upload Matplotlib as described in Using third-party Python packages. Then, create plots in the notebook:
%%spark
import matplotlib.pyplot as plt
import io, base64, json
import matplotlib
# Optional
matplotlib.use('Agg')
# Crucial step: Create the figure here first.
fig = plt.figure()
# Example plot
x = [1, 2, 3, 4]
y = [20, 22, 19, 23]
plt.plot(x, y, marker='o', linestyle='--', color='b')
plt.title("Temperature Change Over Time")
plt.xlabel("Time")
plt.ylabel("Temperature")
# Render the image in the notebook (Livy magic command)
%matplot pltLivy configuration
You can refer to the Apache Livy Configuration, but we recommend against modifying most parameters. The following table lists commonly used configurations.
Parameter | Default | Description |
| true | Specifies whether to enable access control. It is enabled by default, and disabling it is not recommended. |
| 3d | The maximum lifetime of the Livy Server. The default is 3 days. Set the value to |
| 1d | After the Livy Server reaches its maximum lifecycle, this is the maximum time it waits for existing sessions and packaging tasks to finish. No new tasks are accepted during this period. |
| 1h | The maximum time a Livy Server can remain idle without any sessions. |
| 1h | The timeout period for an inactive Spark session. |
| false | When |
| 300s | The duration for which the state of a finished session is retained in Livy's memory for querying. This does not affect session restart. |
| 50 | The maximum number of sessions that can exist simultaneously on a single Livy Server. Default: 50. |
Time parameters usually support s, m, h, and d, such as 30m, 12h, or 3d.