PySpark

Updated at:

The code examples in this topic use Spark 3.x.

Download the template project

If you have already downloaded the project, skip this step. For download instructions, see Use the Spark client in local mode.

The template project includes sample code for spark-1.x, spark-2.x, and spark-3.x. This topic focuses on spark-3.x for demonstration purposes.

Example 1: Read from and write to MaxCompute tables

  • Code details

    Click to view the code

    # -*- coding: utf-8 -*-
    import sys
    from pyspark.sql import SparkSession
    
    try:
      # for python 2
      reload(sys)
      sys.setdefaultencoding('utf8')
    except:
      # python 3 not needed
      pass
    
    if __name__ == '__main__':
      spark = SparkSession.builder\
        .appName("spark sql")\
        .config("spark.sql.broadcastTimeout", 20 * 60)\
        .config("spark.sql.crossJoin.enabled", True)\
        .getOrCreate()
    
      tableName = "mc_test_table"
      ptTableName = "mc_test_pt_table"
      data = [i for i in range(0, 100)]
    
      # Drop Create
      spark.sql("DROP TABLE IF EXISTS %s" % tableName)
      spark.sql("DROP TABLE IF EXISTS %s" % ptTableName)
    
      spark.sql("CREATE TABLE %s (name STRING, num BIGINT)" % tableName)
      spark.sql("CREATE TABLE %s (name STRING, num BIGINT) PARTITIONED BY (pt1 STRING, pt2 STRING)" % ptTableName)
    
      df = spark.sparkContext.parallelize(data, 2).map(lambda s: ("name-%s" % s, s)).toDF("name: string, num: int")
      pt_df = spark.sparkContext.parallelize(data, 2).map(lambda s: ("name-%s" % s, s, "2018", "0601")).toDF("name: string, num: int, pt1: string, pt2: string")
    
      # Write to a standard table
      df.write.insertInto(tableName) # insertInto semantics
      df.writeTo(tableName).overwritePartitions() # insertOverwrite using datasourcev2
    
      # Write to a partitioned table
      # DataFrameWriter cannot write to a specific partition. Use a temporary table and SQL to write to the partition.
      df.createOrReplaceTempView("%s_tmp_view" % ptTableName)
      spark.sql("insert into table %s partition (pt1='2018', pt2='0601') select * from %s_tmp_view" % (ptTableName, ptTableName))
      spark.sql("insert overwrite table %s partition (pt1='2018', pt2='0601') select * from %s_tmp_view" % (ptTableName, ptTableName))
    
      pt_df.write.insertInto(ptTableName) # Dynamic partition, insertInto semantics
      pt_df.write.insertInto(ptTableName, True) # Dynamic partition, insertOverwrite semantics
    
      # Read from a standard table
      rdf = spark.sql("select name, num from %s" % tableName)
      print("rdf count, %s\n" % rdf.count())
      rdf.printSchema()
    
      # Read from a partitioned table
      rptdf = spark.sql("select name, num, pt1, pt2 from %s where pt1 = '2018' and pt2 = '0601'" % ptTableName)
      print("rptdf count, %s" % (rptdf.count()))
      rptdf.printSchema()
  • Submission

    1. You can add dependencies and modify the configuration file based on your Python version. For more information, see PySpark Python versions and dependency support.

    2. Run the following command:

      cd $SPARK_HOME
      bin/spark-submit {{PATH_TO_PROJECT}}/MaxCompute-Spark/spark-3.x/src/main/python/spark_sql.py

Example 2: Alibaba Cloud OSS

  • Code details

    Click to view the code

    # -*- coding: utf-8 -*-
    import sys
    from pyspark.sql import SparkSession
    
    try:
      # for python 2
      reload(sys)
      sys.setdefaultencoding('utf8')
    except:
      # python 3 not needed
      pass
    
    if __name__ == '__main__':
      spark = SparkSession.builder\
        .appName("spark write df to oss")\
        .getOrCreate()
    
      data = [i for i in range(0, 100)]
    
      df = spark.sparkContext.parallelize(data, 2).map(lambda s: ("name-%s" % s, s)).toDF("name: string, num: int")
    
      df.show(n=10)
    
      # write to oss
      pathout = 'oss://<your-bucket>/test.csv'
      df.write.csv(pathout)
  • Submission

    1. You can add dependencies and modify the configuration file based on your Python version. For more information, see PySpark Python versions and dependency support.

    2. You can add the OSS dependency. For more information, see Access Alibaba Cloud OSS.

    3. Run the following command:

      cd $SPARK_HOME
      bin/spark-submit {{PATH_TO_PROJECT}}/MaxCompute-Spark/spark-3.x/src/main/python/spark_oss.py