Spark 2.4.5 usage-MaxCompute(MaxCompute)-阿里云帮助中心

Dedicated configuration for Spark 2.4.5.

Important

We recommend that you use Spark 3 or later.

Submit jobs

To submit a job by using the Spark client, add the following parameters to specify the version. Download the client.

# For enabling Kube mode and eventlog
spark.hadoop.odps.kube.mode=true
spark.hadoop.odps.cupid.data.proxy.enable=true
spark.hadoop.odps.cupid.fuxi.shuffle.enable=true
spark.hadoop.odps.spark.version=spark-2.4.5-odps0.47.0
spark.hadoop.odps.spark.libs.public.enable=true
spark.eventLog.enabled=true
spark.eventLog.dir=/workdir/eventlog/

# For reading from and writing to MaxCompute
spark.sql.catalogImplementation=hive
spark.sql.sources.default=hive

To submit a job by using a DataWorks node, add the following parameter to specify the version.
```
spark.hadoop.odps.spark.version=spark-2.4.5-odps0.47.0
```

Parameters

Parameter	Value	Description
`spark.sql.catalogImplementation`	Set to `hive`.
`spark.sql.sources.default`	Set to `hive`.
`spark.sql.odps.columnarReaderBatchSize`	Default: `4096`	The number of rows per batch for vectorized reads.
`spark.sql.odps.enableVectorizedReader`	Default: `true`	Enables vectorized reads.
`spark.sql.odps.enableVectorizedWriter`	Default: `true`	Enables vectorized writes.
`spark.sql.odps.split.size`	Default: `256m`	Controls the read concurrency on MaxCompute tables. Default split size per partition is 256 MB.
`spark.hadoop.odps.cupid.vnet.capacity`	Default: `256`	Sets the maximum number of instances. We recommend that you set this to `spark.executor.instances + 2`. Otherwise, a `create virtual net failed` error may occur. Configure this parameter in `spark-defaults.conf` or in the `DataWorks` configuration.