MaxCompute Spark node

Permission requirements

To develop jobs, RAM users must be added to the corresponding workspace and granted either the Development or Workspace Administrator role. The Workspace Administrator role includes extensive permissions and should be granted with caution. For more information about how to add a member to a workspace, see Add workspace members.

Limitations

If you encounter an error when submitting a MaxCompute Spark node that uses Spark 3.x, you must purchase and use a serverless resource group. For more information, see Use a serverless resource group.

Before you begin

The MaxCompute Spark node supports running Spark on MaxCompute offline jobs using Java/Scala and Python. The development steps and configuration UI differ for each language. Choose a language based on your business requirements.

Parameters

DataWorks runs Spark on MaxCompute offline jobs in Cluster mode. In Cluster mode, you must specify a custom program entry point main. The corresponding Spark job ends when the main function finishes with a status of Success or Fail. In addition, the configurations in spark-defaults.conf must be added one by one to the MaxCompute Spark node configurations. For example, the number of executors, memory size, and the spark.hadoop.odps.runtime.end.point configuration.

Procedure

1. Create a resource.

   1. Find Resource Management in the left navigation bar of the Data Studio page and click Create. Create a Python resource of the MaxCompute Spark type and name it spark_is_number.py. For more information, see Resource Management. The code is as follows:

      # -*- coding: utf-8 -*-
      import sys
      from pyspark.sql import SparkSession
      try:
          # for python 2
          reload(sys)
          sys.setdefaultencoding('utf8')
      except:
          # python 3 not needed
          pass
      if __name__ == '__main__':
          spark = SparkSession.builder\
              .appName("spark sql")\
              .config("spark.sql.broadcastTimeout", 20 * 60)\
              .config("spark.sql.crossJoin.enabled", True)\
              .config("odps.exec.dynamic.partition.mode", "nonstrict")\
              .config("spark.sql.catalogImplementation", "odps")\
              .getOrCreate()
      def is_number(s):
          try:
              float(s)
              return True
          except ValueError:
              pass
          try:
              import unicodedata
              unicodedata.numeric(s)
              return True
          except (TypeError, ValueError):
              pass
          return False
      print(is_number('foo'))
      print(is_number('1'))
      print(is_number('1.3'))
      print(is_number('-1.37'))
      print(is_number('1e3'))

   2. Save the resource.

2. In the created MaxCompute Spark node, configure the node parameters and the scheduling parameters. For more information, see Parameters.

3. To run the job on a schedule, configure its scheduling properties based on your business requirements. For more information, see Node scheduling configuration.

4. After you configure the node job, deploy the node. For more information, see Deploy nodes/workflows.

5. After the job is deployed, you can go to Operation Center to view the run status of the periodic job. For more information, see Get started with Operation Center.

   Note

   • MaxCompute Spark nodes do not have a run entry point in Data Studio. You must run Spark jobs in Operation Center in the development environment.

   • After the data backfill instance runs successfully, open the tracking URL in the instance run log to view the result.

References

• For more information about developing Spark on MaxCompute jobs for other use cases, see the following topics:

  • Java/Scala example: Spark 1.x

  • Java/Scala example: Spark 2.x

  • Python example: PySpark development

  • Use case: Access a VPC instance from Spark

  • Use case: Access OSS from Spark

• Spark FAQ: Find solutions to common Spark execution issues to accelerate troubleshooting. For more information, see Spark FAQ.

• Spark job diagnosis: MaxCompute provides the Logview tool and the Spark web UI. Use job logs to verify that a job is submitted and running correctly. For more information, see Spark job diagnosis.