Spark 2.3.0 configuration-MaxCompute(MaxCompute)-阿里云帮助中心

Dedicated configuration required for Spark 2.3.0 on MaxCompute.

Important

We recommend that you use Spark 3 or later.

Submit jobs

To submit a job from the Spark client, add the following parameters to specify the version. Download the client.

# Enable Kube mode and event logging
spark.hadoop.odps.kube.mode=true
spark.hadoop.odps.cupid.data.proxy.enable=true
spark.hadoop.odps.cupid.fuxi.shuffle.enable=true
spark.hadoop.odps.spark.version=spark-2.3.0-odps0.47.0
spark.hadoop.odps.spark.libs.public.enable=true
spark.eventLog.enabled=true
spark.eventLog.dir=/workdir/eventlog/

# Read from and write to MaxCompute
spark.sql.catalogImplementation=odps

To submit a job from a DataWorks node, select Spark 2.x and add the following parameter to specify the version.
```
spark.hadoop.odps.spark.version=spark-2.3.0-odps0.47.0
```

Parameters

Parameter	Value	Description
`spark.sql.catalogImplementation`	`odps`
`spark.hadoop.odps.cupid.vectorization.enable`	Recommended: `true`	Enables batch read and write optimization.
`spark.hadoop.odps.input.split.size`	Default: `256`	Controls the read concurrency for MaxCompute tables. Default split size: 256 MB per partition.