Configure Spark SQL task parameters
Spark SQL tasks support different parameters depending on the service type.
Background information
Spark SQL tasks support various service types across different compute engines. For more information, see .
Spark SQL task parameters
Available task parameters vary by service type.
Livy
|
Parameter name |
Description |
|
livy.session.driverMemory |
Memory allocated to the driver process. |
|
livy.session.driverCores |
Number of cores for the driver process. |
|
livy.session.executorMemory |
Memory allocated to each executor process. |
|
livy.session.executorCores |
Number of cores for each executor. |
|
livy.session.numExecutors |
Number of executors to launch for the session. |
|
spark.yarn.queue |
The execution queue. Defaults to Default. |
Kyuubi
|
Parameter name |
Default value |
Description |
|
spark.executor.instances |
1 |
Number of statically allocated executors. |
|
spark.executor.cores |
1 |
Number of cores for each executor. |
|
spark.executor.memory |
1G |
Memory allocated to each executor process. |
|
spark.yarn.am.memory |
512M |
Memory allocated to the YARN Application Master in client mode. |
|
spark.yarn.am.memoryOverhead |
- |
Non-heap memory allocated to the Application Master in client mode. |
|
spark.driver.memory |
1G |
Memory allocated to the driver process. |
|
spark.driver.memoryOverhead |
- |
Non-heap memory allocated to the driver process in cluster mode. |
|
spark.executor.memoryOverhead |
- |
Additional memory for each executor process, covering VM overhead, internal strings, and other native overheads. |
|
spark.yarn.queue |
Default |
The execution queue. Defaults to Default. |
Thrift server, MapReduce (MRS)
|
Parameter name |
Default value |
Description |
|
spark.executor.instances |
1 |
Number of statically allocated executors. |
|
spark.executor.cores |
1 |
Number of cores for each executor. |
|
spark.executor.memory |
1G |
Memory allocated to each executor process. |
|
spark.yarn.am.memory |
512M |
Memory allocated to the YARN Application Master in client mode. |
|
spark.yarn.am.memoryOverhead |
- |
Non-heap memory allocated to the Application Master in client mode. |
|
spark.driver.memory |
1G |
Memory allocated to the driver process. |
|
spark.driver.memoryOverhead |
- |
Non-heap memory allocated to the driver process in cluster mode. |
|
spark.executor.memoryOverhead |
- |
Additional memory for each executor process, covering VM overhead, internal strings, and other native overheads. |