DataWorks Notebook User Guide

Use custom configurations

1. Set SparkConf at startup. For example, set the maximum Livy idle time (config specifies Livy-related parameters) and enable the schema-level syntax switch (spark_conf specifies Spark-related parameters). Other parameters can be configured in the same way.

   %%maxcompute_spark
   {
     "config": {
       # cpu and memory defalut
       "cpu": 1,
       "memory": "4096M",
       "livy.server.max.idle.time": "10m"
     },
     "quota": "XXX",
     "spark_conf": {
       "spark.sql.catalog.odps.enableNamespaceSchema": "true"
     }
   }

2. To modify the configuration, re-run the code above. Note that restarting the Spark session discards the previous session state.

Use OSS storage

Configure OSS access parameters:

%%maxcompute_spark
{
  "spark_conf": {
    "spark.hadoop.odps.cupid.trusted.services.access.list": "<bucket-name>.<oss-endpoint>",
    "spark.hadoop.fs.oss.accessKeyId": "***",
    "spark.hadoop.fs.oss.accessKeySecret": "***",
    "spark.hadoop.fs.oss.endpoint": "<oss-endpoint>"
  }
}

Use third-party Python packages

1. Package dependencies: We recommend that you use pyodps-pack and specify Python 3.11 as the Python version. Livy Spark Session uses Python 3.11 by default, and version consistency avoids compatibility issues. For more information, see PySpark Python Versions and Dependencies.

   # install pyodps
   pip install pyodps
   
   # prepare requirements.txt and pack
   pyodps-pack -r requirements.txt --python-version=3.11 -o <package-name>

2. Upload the package to MaxCompute project:

   --  Run in odpscmd:
   ADD archive /path/to/<package-name> -f;

3. Configure Spark to use the package:

   Note

   • Separate multiple cupid.resources entries with commas (,). For more information about this parameter, see General configurations.

   • Separate multiple PYTHONPATH entries with colons (:). The packages subdirectory in the example is included because pyodps-pack automatically creates this subdirectory. If you use a different packaging method, verify the directory structure before specifying PYTHONPATH. For more information about PYTHONPATH, see Reference user-defined Python packages.

   %%maxcompute_spark
   {
     "spark_conf": {
       "spark.hadoop.odps.cupid.resources": "<your_project>.<package-name>",
       "spark.executorEnv.PYTHONPATH": "./<your_project>.<package-name>/packages",
       "spark.yarn.appMasterEnv.PYTHONPATH": "./<your_project>.<package-name>/packages"
     }
   }

Use JAR or other resource packages

1. Upload the resource (using tar.gz as an example): add archive /path/to/<package-name> -f;

2. Configure Spark to load the resource. Separate multiple resources with commas (,):

   %%maxcompute_spark
   {
     "spark_conf": {
       "spark.hadoop.odps.cupid.resources": "<your_project>.<package-name>"
     }
   }

You can combine this with Python package configurations in the spark_conf parameter.

Advanced configurations

%%maxcompute_spark
{
  "spark_conf": {
    "spark.hadoop.odps.spark.alinux3.enabled": "false"
  }
}

%%spark

import matplotlib.pyplot as plt
import io, base64, json
import matplotlib
# optional
matplotlib.use('Agg')

# A figure must be created here first (critical step).
fig = plt.figure()

# Example plot
x = [1, 2, 3, 4]
y = [20, 22, 19, 23]

plt.plot(x, y, marker='o', linestyle='--', color='b')
plt.title("Temperature Change Over Time")
plt.xlabel("Time")
plt.ylabel("Temperature")

# Render image to Notebook (Livy magic command)
%matplot plt

Livy configurations

The following table lists the configurable Livy parameters: