Spark3.1.1使用-MaxCompute(MaxCompute)-阿里云帮助中心

Configurations specific to Spark 3.1.1 on MaxCompute.

Submit jobs

Spark client: Add the following parameters to specify the version. Download the client.

# For enabling Kube mode and eventlog
spark.hadoop.odps.kube.mode=true
spark.hadoop.odps.cupid.data.proxy.enable=true
spark.hadoop.odps.cupid.fuxi.shuffle.enable=true
spark.hadoop.odps.spark.version=spark-3.1.1-odps0.47.0
spark.hadoop.odps.spark.libs.public.enable=true
spark.eventLog.enabled=true
spark.eventLog.dir=/workdir/eventlog/

# For reading from and writing to MaxCompute
spark.sql.defaultCatalog = odps
spark.sql.catalog.odps = org.apache.spark.sql.execution.datasources.v2.odps.OdpsTableCatalog
spark.sql.sources.partitionOverwriteMode = dynamic
spark.sql.extensions = org.apache.spark.sql.execution.datasources.v2.odps.extension.OdpsExtensions

To run PySpark jobs in cluster mode, add the following parameters to use Python 3.

spark.hadoop.odps.cupid.resources = public.python-3.7.9-ucs4.tar.gz
spark.pyspark.python = ./public.python-3.7.9-ucs4.tar.gz/python-3.7.9-ucs4/bin/python3

To submit jobs by using DataWorks nodes, select Spark 3.x.

Parameters

Parameter	Value	Description
`spark.sql.defaultCatalog`	Set to `odps`.
`spark.sql.catalog.odps`	Set to `org.apache.spark.sql.execution.datasources.v2.odps.OdpsTableCatalog`.
`spark.sql.sources.partitionOverwriteMode`	Set to `dynamic`.
`spark.sql.extensions`	Set to `org.apache.spark.sql.execution.datasources.v2.odps.extension.OdpsExtensions`.
`spark.sql.catalog.odps.enableNamespaceSchema`	Default: `false`	Set to true if schema-level syntax is enabled for the MaxCompute project.
`spark.sql.catalog.odps.enableVectorizedReader`	Default: `true`	Enables vectorized reads.
`spark.sql.catalog.odps.enableVectorizedWriter`	Default: `true`	Enables vectorized writes.
`spark.sql.catalog.odps.splitSizeInMB`	Default: `256`	Controls the read concurrency for MaxCompute tables. Each split defaults to 256 MB.