DataWorks limits file uploads to 10 MB through the console. If your MapReduce job depends on a JAR package or resource file larger than 10 MB, upload it directly through the MaxCompute client and submit only the main function to DataWorks.
Prerequisites
Before you begin, ensure that you have:
-
The MaxCompute client installed and configured. For more information, see Install and configure the MaxCompute client.
Upload the JAR package and run your MapReduce job
Step 1: Upload the JAR package
Run the following command on the MaxCompute client to upload a JAR package that exceeds 10 MB:
-- Upload a JAR package.
add jar C:\test_mr\test_mr.jar -f;Step 2: Verify the upload
Resources uploaded through the MaxCompute client are not displayed on the DataStudio page of the DataWorks console. Run the following command to confirm that the resource was registered:
-- View resources.
list resources;Step 3: Submit the job to DataWorks
DataWorks runs the MapReduce job on the machine where the MaxCompute client is installed. Because of this, submit only the Mapper and Reducer that contain the main function—not third-party dependencies. Store your resources in the wc_in directory of the MaxCompute client and reference them using -resources:
jar
-resources test_mr.jar,test_ab.jar -- A file can be referenced after it is registered on the MaxCompute client.
-classpath test_mr.jar -- Submit only the Mapper and Reducer that contain the main function.Third-party dependencies do not need to be submitted to DataWorks. Only include the JAR that contains the main function in -classpath.