Run EMR PySpark with the Python3 kernel

Prerequisites

Download the sample notebook file spark_magic_example.ipynb to your local directory.

Procedure

1. Access the JupyterHub web UI. For more information, see Manage JupyterHub.
2. On the JupyterHub web UI, under Notebook, click Python3.
3. Click the icon to upload the sample notebook file.

   In the spark_magic_example.ipynb notebook, you can view the provided PySpark and SQL magic commands.

   • PySpark command parameters
     • set_spark_opts
       Sets the Spark runtime parameters. You can configure any parameter supported by the Spark version in your environment. It accepts a dictionary. For example:

       conf = {
           'spark.executor.instances': '3',
           'spark.executor.memory': '2g',
       }
       
       %set_spark_opts $conf

     • get_spark_opts
       Displays the Spark runtime parameters. For example:

       %get_spark_opts

     • get_spark_session
       Starts a Spark job on the E-MapReduce cluster based on the specified runtime parameters and returns the Spark session. This parameter accepts a string that specifies the Spark job name. For example:

       spark, sc = %get_spark_session test_application

       Important Each notebook supports only one Spark session. If you try to create another one, the existing session is returned.
     • stop_spark_session

       Terminates the Spark job.

   • SQL command parameters
     • In a %%sql cell, you can write multiple SQL statements separated by semicolons (;). For example:

       %%sql
       
       create table if not exists test_table_1  (a string, b int);
       insert into test_table_1 values("abc", 1), ("def", 2);
       select a, sum(b) from test_table_1 group by a

     • In a %%sql cell, you can use the -o parameter to assign the result to a specified variable as a pandas DataFrame. If you omit this parameter, the cell directly displays the result of the last SQL statement. For example:

       %%sql -o df  # Assigns the result to the df variable.
       
       select a, sum(b) from test_table group by a

     • In a %%sql cell, you can use variables from other cells. For example:

       %%sql
       
       select * from test_table where a = '{x}' # The variable x is defined in another cell. This technique is often used with schedulers like Airflow.

4. Click the icon at the top of the notebook to run the current cell.