Develop MapReduce

Write MapReduce

1. In the Project tool window, right-click the source directory of the module (src > main > java) and select New > MaxCompute Java.

   

2. Enter a Name, select Driver as the class type, and then press Enter.

   

   • Name: The name of the MaxCompute Java class. If you have not created a package, enter the name in the packagename.classname format to automatically create a package.

   • Select the class type: Driver, Mapper, or Reducer.

     Note

     Select the appropriate class type:

     • Driver: The driver class for the MapReduce job. It configures and submits the job. You can specify the Mapper and Reducer classes and other configuration details within the driver. It is the entry point for the job.

     • Mapper: The first stage of MapReduce data processing. It processes each input record and generates an intermediate key-value pair.

     • Reducer: Receives the intermediate key-value pairs from the Mapper, processes them, and produces the final output. The output is then saved to a MaxCompute table.

3. After creating the class, write your Java code in the editor.

   MaxCompute Studio automatically populates the Java template with framework code. You only need to configure settings such as the input table, output table, and the Mapper and Reducer classes.

   For a MapReduce Java program example, see WordCount example.

   

Debug with a local run

Use a local run to test your MapReduce program and verify that the output meets your expectations.

1. Right-click the completed Java file and select Run.

2. In the Run/Debug Configurations dialog box, select the MaxCompute project for this run.

   

3. Click OK to start the run.

   Note

   • A local run reads the table data specified in warehouse as input. You can view the log output in the console.

   • For more information about the warehouse directory, see the warehouse directory.

Debug with unit testing

Package and upload

After debugging your program, package it into a JAR file and upload it to MaxCompute as a resource. For more information, see Package, upload, and register a Java program.

Run MapReduce

Run the MapReduce program on the MaxCompute client.

1. In the left-side navigation pane, click Project Explorer.

2. Right-click the project name and select Open in Console.

3. In the Console pane, run the following command to run the MapReduce job.

   For more information about commands, see JAR commands.

   jar -resources wordcount.jar -classpath D:\odps\clt\wordcount.jar com.aliyun.odps.examples.mr.WordCount wc_in wc_out;