Spark3.4.2配置-MaxCompute(MaxCompute)-阿里云帮助中心

Dedicated configurations required for Spark 3.4.2 & 3.5.2.

Submit jobs

Use cluster mode

Submit jobs by using the Spark client.

Add the following parameters to specify the version. Download the client: Spark 3.4.2 or Spark 3.5.2.

# For enabling Kube mode and eventlog
spark.hadoop.odps.kube.mode=true
spark.hadoop.odps.cupid.data.proxy.enable=true
spark.hadoop.odps.cupid.fuxi.shuffle.enable=true

## for spark 3.4.2
spark.hadoop.odps.spark.version=spark-3.4.2-odps0.48.0

## for spark 3.5.2
spark.hadoop.odps.spark.version=spark-3.5.2-odps0.49.0
spark.hadoop.odps.spark.libs.public.enable=true
spark.eventLog.enabled=true
spark.eventLog.dir=/workdir/eventlog/

# For readding from and writing to MaxCompute
spark.sql.defaultCatalog=odps
spark.sql.catalog.odps=org.apache.spark.sql.execution.datasources.v2.odps.OdpsTableCatalog
spark.sql.sources.partitionOverwriteMode=dynamic
spark.sql.extensions=org.apache.spark.sql.execution.datasources.v2.odps.extension.OdpsExtensions

Submit jobs by using DataWorks nodes. Add the following parameters to specify the version.

## for spark 3.4.2
spark.hadoop.odps.spark.version=spark-3.4.2-odps0.48.0

## for spark 3.5.2
spark.hadoop.odps.spark.version=spark-3.5.2-odps0.49.0

Parameter configuration

Parameter	Value	Description
`spark.sql.defaultCatalog`	Set the value to `odps`.
`spark.sql.catalog.odps`	Set the value to `org.apache.spark.sql.execution.datasources.v2.odps.OdpsTableCatalog`.
`spark.sql.sources.partitionOverwriteMode`	Set the value to `dynamic`.
`spark.sql.extensions`	Set the value to `org.apache.spark.sql.execution.datasources.v2.odps.extension.OdpsExtensions`.
`spark.sql.catalog.odps.enableNamespaceSchema`	Default value: `false`	Set this parameter to true if the schema-level syntax switch is enabled for your MaxCompute project.
`spark.sql.catalog.odps.enableVectorizedReader`	Default value: `true`	Enables vectorized reads.
`spark.sql.catalog.odps.enableVectorizedWriter`	Default value: `true`	Enables vectorized writes.
`spark.sql.catalog.odps.splitSizeInMB`	Default value: `256`	Controls the read concurrency for MaxCompute tables. The default split size is 256 MB per partition.
`spark.sql.catalog.odps.tableReadProvider`	Default value: `v1`	Set this parameter to tunnel when you use the `local` mode.
`spark.sql.catalog.odps.tableWriteProvider`	Default value: `v1`	Set this parameter to tunnel when you use the `local` mode.
`spark.hadoop.odps.spark.alinux3.enabled`	Default value: `false`	Uses the Alinux3 base image with Python 3.11 in cluster mode.
`spark.hadoop.odps.native.engine.enable`	Default value: `false`	Enables Native Engine to accelerate computing in cluster mode. Native Engine uses the Alinux3 base image by default.