Spark SQL tasks support different parameters depending on the service type.
Background information
Spark SQL tasks support various service types across different compute engines. For more information, see .
Spark SQL task parameters
Available task parameters vary by service type.
Livy
|
Parameter name |
Description |
|
livy.session.driverMemory |
Memory allocated to the driver process. |
|
livy.session.driverCores |
Number of cores for the driver process. |
|
livy.session.executorMemory |
Memory allocated to each executor process. |
|
livy.session.executorCores |
Number of cores for each executor. |
|
livy.session.numExecutors |
Number of executors to launch for the session. |
|
spark.yarn.queue |
The execution queue. Defaults to Default. |
Kyuubi
|
Parameter name |
Default value |
Description |
|
spark.executor.instances |
1 |
Number of statically allocated executors. |
|
spark.executor.cores |
1 |
Number of cores for each executor. |
|
spark.executor.memory |
1G |
Memory allocated to each executor process. |
|
spark.yarn.am.memory |
512M |
Memory allocated to the YARN Application Master in client mode. |
|
spark.yarn.am.memoryOverhead |
- |
Non-heap memory allocated to the Application Master in client mode. |
|
spark.driver.memory |
1G |
Memory allocated to the driver process. |
|
spark.driver.memoryOverhead |
- |
Non-heap memory allocated to the driver process in cluster mode. |
|
spark.executor.memoryOverhead |
- |
Additional memory for each executor process, covering VM overhead, internal strings, and other native overheads. |
|
spark.yarn.queue |
Default |
The execution queue. Defaults to Default. |
Thrift server, MapReduce (MRS)
|
Parameter name |
Default value |
Description |
|
spark.executor.instances |
1 |
Number of statically allocated executors. |
|
spark.executor.cores |
1 |
Number of cores for each executor. |
|
spark.executor.memory |
1G |
Memory allocated to each executor process. |
|
spark.yarn.am.memory |
512M |
Memory allocated to the YARN Application Master in client mode. |
|
spark.yarn.am.memoryOverhead |
- |
Non-heap memory allocated to the Application Master in client mode. |
|
spark.driver.memory |
1G |
Memory allocated to the driver process. |
|
spark.driver.memoryOverhead |
- |
Non-heap memory allocated to the driver process in cluster mode. |
|
spark.executor.memoryOverhead |
- |
Additional memory for each executor process, covering VM overhead, internal strings, and other native overheads. |