Referencing external files
This topic describes how to reference external files in MaxCompute Spark jobs. You can upload files either using Spark parameters or using MaxCompute resources.
Use Cases
Your job needs to read a configuration file.
Your job needs additional JAR packages or Python libraries.
Upload Files
You can upload files in two ways:
Upload files using Spark parameters
MaxCompute Spark supports native Spark community parameters such as --jars and --py-files. When you submit a job, use these parameters to upload files. The files are uploaded to the user's working directory when the job runs.
Upload using the Spark client: Use the
spark-submitcommand-line parameter directly.Parameter descriptions
--jars:Uploads the specified JAR packages to the current working directory of the Driver and Executors. You can separate multiple files with commas. These JAR packages are added to the Classpath of both the Driver and Executors. In your Spark job, reference them as
./your_jar_name.--filesand--py-files:Uploads the specified regular files or Python files to the current working directory of the Driver and Executors. You can separate multiple files with commas. In your Spark job, reference them as
./your_file_name.--archives:Behavior differs slightly from the open-source Spark version. You can separate multiple files with commas. You can specify archives in the format
xxx#yyy. This extracts the archive (such as a .zip file) into a subdirectory of the current working directory on the Driver and Executors. To extract archive contents directly into the current directory, use thespark.hadoop.odps.cupid.resourcesconfiguration.For example, if you specify
xx.zip#yy, reference the archive contents as./yy/xx/. If you specify onlyxx.zip, reference the contents as./xx.zip/xx/.
Upload using DataWorks: Add required resources to your task. For more information, see Using DataWorks Spark.
Upload files using MaxCompute resources
MaxCompute Spark provides the spark.hadoop.odps.cupid.resources parameter. You can use it to reference resources stored in MaxCompute. These resources are uploaded to the working directory when the job runs.
How to use
You can upload files using the MaxCompute client. Each file can be up to 500 MB.
In your Spark job configuration, add the
spark.hadoop.odps.cupid.resourcesparameter. Use the format<projectname>.<resourcename>. You can separate multiple files with commas.
spark.hadoop.odps.cupid.resources Parameter description
This configuration item specifies the MaxCompute resources required for the job. It must be configured in
spark-default.confor in DataWorks configuration settings to take effect. Do not set it in your code.Example configuration:
spark.hadoop.odps.cupid.resources=public.python-python-2.7-ucs4.zip,public.myjar.jarUsage notes:
The specified resources are downloaded to the current working directory of the Driver and Executors. After download, each resource uses the default name
<projectname>.<resourcename>.Rename files:
You can specify a new name in the configuration using
<projectname>.<resourcename>:<newresourcename>. For example:spark.hadoop.odps.cupid.resources=public.myjar.jar:myjar.jar
Reference files in code
After uploading files to the current working directory using either method above, read them in your code as shown in this example:
val targetFile = "filename"
val file = Source.fromFile(targetFile)
for (line <- file.getLines)
println(line)
file.close