Lindorm Spark node

Upload a JAR package

Upload the sample JAR package to LindormDFS and copy its storage path. This path is required later to configure the node.

1. Prepare the sample JAR package.

   Download the sample spark-examples_2.12-3.3.0.jar package to your computer.

2. Upload the JAR package to LindormDFS.

   1. Log on to the Lindorm console. In the Instances list for the corresponding region, find the Lindorm instance that you created.

   2. Click the instance Name to open the Instance Details page.

   3. In the left navigation bar, click Compute Engines to go to the Compute Engines page.

   4. On the Job Management tab, click Upload resources to open the Upload resources page.

   5. Click the dotted-line box, find the JAR package that you downloaded, and then click Open.

   6. Click Upload to upload the JAR package.

3. Copy the storage path of the sample JAR package.

   On the Job Management tab, find the JAR package that you uploaded. It appears under the Upload Resource button. Click the icon to the left of the file name to copy the storage path of the JAR package in LindormDFS.

Configure node content

Configure the Lindorm Spark node with the following parameters.

Upload a Python resource

Upload the sample Python file to LindormDFS and copy its storage path. This path is required later to configure the node.

1. Create a Python file.

   Save the following Python script as a local file named pi.py.

   import sys
   from random import random
   from operator import add
   
   from pyspark.sql import SparkSession
   
   if __name__ == "__main__":
       """
           Usage: pi [partitions]
       """
       spark = SparkSession\
           .builder\
           .appName("PythonPi")\
           .getOrCreate()
   
       partitions = int(sys.argv[1]) if len(sys.argv) > 1 else 2
       n = 100000 * partitions
   
       def f(_: int) -> float:
           x = random() * 2 - 1
           y = random() * 2 - 1
           return 1 if x ** 2 + y ** 2 <= 1 else 0
   
       count = spark.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
       print("Pi is roughly %f" % (4.0 * count / n))
   
       spark.stop()
   

2. Upload the Python file to LindormDFS.

   1. Log on to the Lindorm console. In the Instances list for the corresponding region, find the Lindorm instance that you created.

   2. Click the instance Name to open the Instance Details page.

   3. In the left navigation bar, click Compute Engines to go to the Compute Engines page.

   4. On the Job Management tab, click the Upload resources button to open the Upload resources page.

   5. Click the dotted-line box, find the Python file that you created, and then click Open.

   6. Click Upload to upload the Python file.

3. Copy the storage path of the sample Python file.

   On the Job Management tab, find the Python file that you uploaded. It appears under the Upload Resource button. Click the icon to the left of the file name to copy the storage path of the Python file in LindormDFS.

Configure node content

Configure the Lindorm Spark node with the following parameters.