Using open source ML libraries

Step 1: Download JAR files

Download all MMLSpark JAR files on your local client:

1. Download a Spark client to your local machine.

2. Configure spark-defaults.conf and add the following parameters:

   spark.master = local[*]
   spark.jars.repositories=https://mmlspark.azureedge.net/maven

3. Run the following command in Local mode:

   $SPARK_HOME/bin/pyspark --packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1

4. The JAR files are typically downloaded to the following directory: $HOME/.ivy2/jars

5. Compress all JAR files into a ZIP archive:

   cd $HOME/.ivy2/jars
   zip -r mml_spark.zip .

Step 2: Modify spark-defaults.conf

spark.executor.extraClassPath=./mml_spark.zip/*
spark.driver.extraClassPath=./mml_spark.zip/*

Step 3: Submit a job

Submit the job to the cluster in Yarn-cluster mode. Make sure to include --py-files:

./bin/spark-submit --archives mml_spark.zip \
--py-files mml_spark/com.microsoft.ml.spark_mmlspark_2.11-1.0.0-rc1.jar,mml_spark/com.microsoft.ml.lightgbm_lightgbmlib-2.3.100.jar \
spark_mml.py

Step 1: Package Python

1. Install Analytics Zoo:

   ./bin/pip3 install analytics-zoo -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com

2. Uninstall PySpark because MaxCompute Spark already includes PySpark:

   ./bin/pip3 uninstall pyspark

3. Package the environment as an archive. The following example uses the name python-3.6.14-big-dl.tar.gz.

Step 2: Upload the Python package to MaxCompute Resource

Upload the Python package to MaxCompute Resource.

Step 3: Prepare JAR files

Copy the following three JAR files from the paths listed below:

$python_home/lib/python3.6/site-packages/zoo/share/lib
  - analytics-zoo-bigdl_0.13.0-spark_2.4.6-0.11.1-jar-with-dependencies.jar

$python_home/lib/python3.6/site-packages/bigdl/share/lib
  - bigdl-0.13.0-jar-with-dependencies.jar
  - bigdl-0.13.0-python-api.zip

Step 4: Repackage BigDL to resolve log4j class conflicts

1. Find the BigDL version that corresponds to Analytics Zoo. For example, Analytics Zoo 0.11.0 corresponds to BigDL 0.13.0.

2. Download the BigDL source code and switch to the corresponding branch:

   git clone https://github.com/intel-analytics/BigDL.git
   git checkout branch-0.13

3. Build the package:

   cd BigDL/spark/dl/
   mvn clean package -DskipTests

4. Replace bigdl-0.13.0-jar-with-dependencies.jar from Step 3 with bigdl-0.13.1-SNAPSHOT-jar-with-dependencies.jar generated in the target directory.

Step 5: Configure the Python package

Add the following configuration to spark-defaults.conf:

spark.hadoop.odps.cupid.resources = [projectname].python-3.6.14-big-dl.tar.gz
spark.pyspark.python = ./[projectname].python-3.6.14-big-dl.tar.gz/python-3.6.14-big-dl/bin/python3

Step 6: Submit a job

Include the JAR files from Step 3 and Step 4 when you submit the job:

./bin/spark-submit \
--jars analytics-zoo-bigdl_0.13.0-spark_2.4.6-0.11.1-jar-with-dependencies.jar,bigdl-0.13.1-SNAPSHOT-jar-with-dependencies.jar,bigdl-0.13.0-python-api.zip \
spark_test.py