Manage SQL sessions

Create an SQL session

After you create an SQL session, you can select it when you create an SQL job.

1. Go to the Sessions page.

   1. Log on to the EMR console.

   2. In the left-side navigation pane, choose EMR Serverless > Spark.

   3. On the Spark page, click the name of the target workspace.

   4. On the EMR Serverless Spark page, click Sessions in the left-side navigation pane.

2. On the SQL Session page, click Connect to SQL Session.

3. On the Create SQL Session page, configure the following parameters and click create.

   Important

   We recommend that you set the maximum concurrency of the selected resource queue to at least the amount of resources required by the notebook session. This value is displayed on the console.

   Parameter	Description
   Name	The name of the SQL session. The name must be 1 to 64 characters in length and can contain letters, digits, hyphens (-), underscores (_), and spaces.
   Resource Queue	The resource queue for the SQL session. You can select a queue designated for development or one shared between development and production. For more information about queues, see Manage resource queues.
   Engine Version	The engine version for the SQL session. For more information, see Engine versions.
   Use Fusion Acceleration	Fusion can accelerate Spark workloads and reduce the total cost of jobs. For billing information, see Product Billing. For more information about the Fusion engine, see Fusion engine.
   Automatic Stop	Enabled by default. You can specify a custom idle timeout. The system automatically stops the SQL session after it remains idle for the specified period.
   Normal Network Connection	An existing network connection for accessing data sources in a VPC or external services. For more information, see Network connectivity between EMR Serverless Spark and other VPCs.
   spark.driver.cores	The number of CPU cores for the driver process. Default value: 1.
   spark.driver.memory	The amount of memory for the driver process. Default value: 3.5 GB.
   spark.executor.cores	The number of CPU cores for each executor process. Default value: 1.
   spark.executor.memory	The amount of memory for each executor process. Default value: 3.5 GB.
   spark.executor.instances	The number of executors. Default value: 2.
   Dynamic Resource Allocation	Disabled by default. When enabled, configure the following parameters: Minimum Number of Executors: The default value is 2. Maximum Number of Executors: If spark.executor.instances is not set, the default value is 10.
   More Memory Configurations	spark.driver.memoryOverhead: The non-heap memory available for the driver. If this parameter is not set, Spark automatically allocates a value based on the default, which is max(384 MB, 10% * spark.driver.memory). spark.executor.memoryOverhead: The non-heap memory available for each executor. If this parameter is not set, Spark automatically allocates a value based on the default, which is max(384 MB, 10% * spark.executor.memory). spark.memory.offHeap.size: The amount of off-heap memory available to Spark. The default value is 1 GB. This parameter takes effect only when spark.memory.offHeap.enabled is set to true. When the Fusion engine is used, this feature is enabled by default with 1 GB of off-heap memory.
   Spark Configuration	Custom Spark configuration properties. Use spaces to separate key-value pairs. Example: spark.sql.catalog.paimon.metastore dlf.

   After you create an SQL session, its status is initially Starting. The session is ready when its status changes to Running. On the SQL Session page, you can stop, edit, or delete sessions.

View execution records

After a job completes, you can view its execution history.

1. On the SQL Sessions page, click the name of the desired session.

2. Click the Execution Records tab.

   On this tab, you can view details for each execution, such as the run ID, start time, and a link to the Spark UI.

   

Related topics

• For queue-related operations, see Manage resource queues.

• For information about roles and permissions for sessions, see Manage users and roles.

• For a complete example of the sql job development workflow, see Spark SQL quick start.