Local mode

Step 1: Download and decompress the Spark client

1. Click this link or run the following commands in a terminal to download and extract the Spark client. This example uses Spark client version 3.1.1-odps0.47.0.

   wget https://odps-repo.oss-cn-hangzhou.aliyuncs.com/spark/3.1.1-odps0.47.0/spark-3.1.1-odps0.47.0-client.tar.gz
   tar zxvf spark-3.1.1-odps0.47.0-client.tar.gz

2. Verify that the client is downloaded and extracted to the correct directory.

   

Step 2: Create and modify the odps.conf configuration file

In the spark-3.1.1-odps0.47.0-client/conf directory, create an odps.conf file:

touch odps.conf
vim odps.conf

Add the following configurations to the file:

odps.project.name = [Replace with your actual project]
odps.end.point = [Replace with your actual endpoint]
odps.access.id = [Replace with your actual access ID]
odps.access.key = [Replace with your actual access key]

Step 3: Open the sample project in IntelliJ IDEA

1. On the IntelliJ IDEA welcome screen, click Get from VCS and paste the following repository URL: https://github.com/aliyun/MaxCompute-Spark.git.

   

2. If a security warning prompts you to trust the project, click Trust Project.

3. After the repository is cloned, IntelliJ IDEA automatically opens the project at its root directory, MaxCompute-Spark. Close the current project and reopen it from the correct subdirectory.

   1. Close the current project:

      • From the menu bar, choose File > Close Project to return to the welcome screen.

   2. Open the project from the subdirectory:

      • On the welcome screen, click Open.

      • In the file browser that opens, navigate to the folder that you cloned.

      • In the MaxCompute-Spark folder, select the spark-3.x subdirectory and click Open.

   

Step 4: Add the MaxCompute Spark Jars dependency

1. Navigate to File > Project Structure.

   

2. In the left-side navigation pane, select Modules, click the + icon, and then select JARs or Directories.

   

3. Select the spark-3.1.1-odps0.47.0-client/jars directory that you extracted in Step 1.

   

4. Click OK.

Step 5: Add the ODPS_CONF_FILE environment variable

1. In the upper-right corner of IntelliJ IDEA, click Edit Configurations.

   

2. Set environment variables

   

3. Add ODPS_CONF_FILE. Set Name to ODPS_CONF_FILE and Value to the path to the odps.conf file that you created in Step 2.

   

   Click OK.

Step 6: Run the JavaSparkSQL.java task

1. On line 48, add the following code to configure local mode:

   .config("spark.master","local[*]")

   

2. On line 113, add the following code to manually add a partition:

   spark.sql("ALTER TABLE " + ptTableName + " ADD IF NOT EXISTS PARTITION (pt1='2018', pt2='0601')");

   

3. Comment out the code on line 118. local mode does not support overwriting dynamic partitions.

   ptDf.write().mode("overwrite").insertInto(ptTableName); 

   

4. Run the task.

   Click to expand and view details

   +-------+---+
   |   name|num|
   +-------+---+
   | name-0|  0|
   | name-1|  1|
   | name-2|  2|
   | name-3|  3|
   | name-4|  4|
   | name-5|  5|
   | name-6|  6|
   | name-7|  7|
   | name-8|  8|
   | name-9|  9|
   |name-10| 10|
   |name-11| 11|
   |name-12| 12|
   |name-13| 13|
   |name-14| 14|
   |name-15| 15|
   |name-16| 16|
   |name-17| 17|
   |name-18| 18|
   |name-19| 19|
   +-------+---+
   only showing top 20 rows

Step 1: Download and decompress the Spark client

1. Click this link or run the following commands in a terminal to download and extract the Spark client. This example uses Spark client version 3.1.1-odps0.47.0.

   wget https://odps-repo.oss-cn-hangzhou.aliyuncs.com/spark/3.1.1-odps0.47.0/spark-3.1.1-odps0.47.0-client.tar.gz
   tar zxvf spark-3.1.1-odps0.47.0-client.tar.gz

2. Verify that the client is downloaded and extracted to the correct directory.

   

Step 2: Create and modify the odps.conf configuration file

In the spark-3.1.1-odps0.47.0-client/conf directory, create an odps.conf file:

touch odps.conf
vim odps.conf

Add the following configurations to the file:

odps.project.name = [Replace with your actual project]
odps.end.point = [Replace with your actual endpoint]
odps.access.id = [Replace with your actual access ID]
odps.access.key = [Replace with your actual access key]

Step 3: Modify the spark-defaults.conf configuration file

Open the spark-defaults.conf file located in the spark-3.1.1-odps0.47.0-client/conf directory.

cd spark-3.1.1-odps0.47.0-client/conf
vim spark-defaults.conf

# Comment out the following configurations
#spark.master = yarn
#spark.submit.deployMode = cluster
#spark.eventLog.enabled = true

# Add the following configuration
spark.master = local[*]

Step 4: Add the HADOOP_CONF_DIR environment variable

export HADOOP_CONF_DIR=/PATH/TO/spark-3.1.1-odps0.47.0-client/conf

Step 5: Open the PySpark shell

cd spark-3.1.1-odps0.47.0-client
./bin/pyspark

Step 6: Run Spark SQL

Query a table in the MaxCompute project specified in the configuration file.

spark.sql("SELECT * FROM xxxx").show(); 

spark.sql("SELECT * FROM emp").show();

// Returned result
+-----+------+---------+----+-------------------+-------+-------+------+--------+
|empno| ename|      job| mgr|           hiredate|    sal|   comm|deptno|      ds|
+-----+------+---------+----+-------------------+-------+-------+------+--------+
| 7369| SMITH|    CLERK|7902|1980-12-17 00:00:00| 800.00|   null|    20|20251229|
| 7499| ALLEN| SALESMAN|7698|1981-02-20 00:00:00|1600.00| 300.00|    30|20251229|
| 7521|  WARD| SALESMAN|7698|1981-02-22 00:00:00|1250.00| 500.00|    30|20251229|
| 7566| JONES|  MANAGER|7839|1981-04-02 00:00:00|2975.00|   null|    20|20251229|
| 7654|MARTIN| SALESMAN|7698|1981-09-28 00:00:00|1250.00|1400.00|    30|20251229|
| 7698| BLAKE|  MANAGER|7839|1981-05-01 00:00:00|2850.00|   null|    30|20251229|
| 7782| CLARK|  MANAGER|7839|1981-06-09 00:00:00|2450.00|   null|    10|20251229|
| 7788| SCOTT|  ANALYST|7566|1987-04-19 00:00:00|3000.00|   null|    20|20251229|
| 7839|  KING|PRESIDENT|null|1981-11-17 00:00:00|5000.00|   null|    10|20251229|
| 7844|TURNER| SALESMAN|7698|1981-09-08 00:00:00|1500.00|   0.00|    30|20251229|
| 7876| ADAMS|    CLERK|7788|1987-05-23 00:00:00|1100.00|   null|    20|20251229|
| 7900| JAMES|    CLERK|7698|1981-12-03 00:00:00| 950.00|   null|    30|20251229|
| 7902|  FORD|  ANALYST|7566|1981-12-03 00:00:00|3000.00|   null|    20|20251229|
| 7934|MILLER|    CLERK|7782|1982-01-23 00:00:00|1300.00|   null|    10|20251229|
| 7948|JACCKA|    CLERK|7782|1981-04-12 00:00:00|5000.00|   null|    10|20251229|
| 7956| WELAN|    CLERK|7649|1982-07-20 00:00:00|2450.00|   null|    10|20251229|
| 7957|TEBAGE|    CLERK|7748|1982-12-30 00:00:00|1300.00|   null|    10|20251229|
+-----+------+---------+----+-------------------+-------+-------+------+--------+