PySpark
The code examples in this topic use Spark 3.x.
Download the template project
If you have already downloaded the project, skip this step. For download instructions, see Use the Spark client in local mode.
The template project includes sample code for spark-1.x, spark-2.x, and spark-3.x. This topic focuses on spark-3.x for demonstration purposes.
Example 1: Read from and write to MaxCompute tables
Submission
You can add dependencies and modify the configuration file based on your Python version. For more information, see PySpark Python versions and dependency support.
Run the following command:
cd $SPARK_HOME bin/spark-submit {{PATH_TO_PROJECT}}/MaxCompute-Spark/spark-3.x/src/main/python/spark_sql.py
Example 2: Alibaba Cloud OSS
Submission
You can add dependencies and modify the configuration file based on your Python version. For more information, see PySpark Python versions and dependency support.
You can add the OSS dependency. For more information, see Access Alibaba Cloud OSS.
Run the following command:
cd $SPARK_HOME bin/spark-submit {{PATH_TO_PROJECT}}/MaxCompute-Spark/spark-3.x/src/main/python/spark_oss.py