Spark Launcher

Step 1: Download and extract the Spark client

1. Click this link or run the following commands in a terminal to download and extract the Spark client. This example uses Spark client version 3.1.1-odps0.47.0.

   wget https://odps-repo.oss-cn-hangzhou.aliyuncs.com/spark/3.1.1-odps0.47.0/spark-3.1.1-odps0.47.0-client.tar.gz
   tar zxvf spark-3.1.1-odps0.47.0-client.tar.gz

2. Verify that the client is downloaded and extracted to the correct directory.

   

Note: The client you downloaded here is the actual runtime client. The spark_launcher version you add in your pom.xml file is used only to launch the task.

Step 2: Set environment variables

# Recommend JDK 1.8
export JAVA_HOME={{JAVA_PATH}}/jdk
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH

# Download the MaxCompute Spark client mentioned above and extract it to any local path.
export SPARK_HOME={{SPARK_PATH}}
export PATH=$SPARK_HOME/bin:$PATH

export HADOOP_CONF_DIR=$SPARK_HOME/conf

Step 3: Configure settings in your code

To clone the repository, see Open the sample project in IntelliJ IDEA. The file location and code segment appear in the following figure:

package com.aliyun.odps.spark.examples.sparksql;
import org.apache.spark.launcher.SparkLauncher;
public class JavaSparkLauncherExample {
    public static void main(String[] args) throws Exception {
        // set spark home from env or args
        String sparkHome = System.getenv("SPARK_HOME");
        if (sparkHome == null) {
            // please set your own spark client path
            sparkHome = "/path/to/spark";
        }
        // please set your own AppResource, PropertiesFile and MainClass
        // you can also use setConf to replace setPropertiesFile
        SparkLauncher launcher = new SparkLauncher()
                .setAppName("Spark Launcher Example")
                .setSparkHome(sparkHome)
                .setAppResource("./target/spark-examples_2.12-1.0.0-SNAPSHOT.jar")
                .setPropertiesFile(sparkHome + "/conf/spark-defaults.conf")
                .setConf("spark.launcher.childProcLoggerName", "launcherLogger")
                .setMainClass("com.aliyun.odps.spark.examples.SparkPi");
        launcher.launch().waitFor();
    }
}

• setAppResource(path/to/mainJar): Set the main JAR file as mainJar.

• Use setPropertiesFile or setConf to set the configuration file propertiesFile.

  A sample conf/spark-defaults.conf is included in the MaxCompute Spark client download. For more options, see Spark Configuration.

• spark.launcher.childProcLoggerName: We recommend setting this to display logs.

• setMainClass: Set the main class to run as mainClass.

• Call launcher.launch().waitFor(), which, under the hood, uses a child process to execute spark-submit.