Develop MapReduce

Write MapReduce

1. In the Project pane, right-click the source code directory of the module (that is, src > main > java), and select New > MaxCompute Java.

2. Enter a Name, select Driver as the class type, and then press Enter.

   • Name: The name of the MaxCompute Java class. If you have not created a package, enter the name in the packagename.classname format to automatically create a package.

   • Select the class type: Driver, Mapper, or Reducer.

     Note

     Select the appropriate class type:

     • Driver: The driver class for the MapReduce job. It configures and submits the job. You can specify the Mapper and Reducer classes and other configuration details within the driver. It is the entry point for the job.

     • Mapper: The first stage of MapReduce data processing. It processes each input record and generates an intermediate key-value pair.

     • Reducer: Receives the intermediate key-value pairs from the Mapper, processes them, and produces the final output. The output is then saved to a MaxCompute table.

3. After creating the class, write your Java code in the editor.

   MaxCompute Studio automatically populates the Java template with framework code. You only need to configure settings such as the input table, output table, and the Mapper and Reducer classes.

   For a MapReduce Java program example, see WordCount example.

   package mymr.myudf;
   
   import ...
   
   public class HelloDriver {
   
       public static void main(String[] args) throws OdpsException {
   
           JobConf job = new JobConf();
   
           // TODO: specify map output types
           job.setMapOutputKeySchema(SchemaUtils.fromString(?));
           job.setMapOutputValueSchema(SchemaUtils.fromString(?));
   
           // TODO: specify input and output tables
           InputUtils.addTable(TableInfo.builder().tableName(?).build(), job);
           OutputUtils.addTable(TableInfo.builder().tableName(?).build(), job);
   
           // TODO: specify a mapper
           job.setMapperClass(?);
           // TODO: specify a reducer
           job.setReducerClass(?);
   
           RunningJob rj = JobClient.runJob(job);
           rj.waitForCompletion();
   
       }
   }

Debug with a local run

Use a local run to test your MapReduce program and verify that the output meets your expectations.

1. Right-click the completed Java file and select Run.

2. In the Run/Debug Configurations dialog box, select the MaxCompute project for this run.

   In the Run/Debug Configurations dialog box, select WordCountTest under JUnit in the left-side pane. In the configuration pane on the right, set Test kind to Class, enter com.aliyun.odps.examples.mr.test.WordCountTest in the Class field, and set Use classpath of module to MyFristModule.

3. Click OK to start running.

   Note

   • The system reads data from the specified table in warehouse as the input during the local run. You can view the log output in the console.

   • For more information about the warehouse directory, see warehouse directory.

Debug with unit testing

package com.aliyun.odps.examples.mr.test;

import ...

public class WordCountTest extends M...

    // Define the schemas of the input and output tables
    private final static String INPUT_...
    private final static String OUTPUT_...
    private JobConf job;

    public WordCountTest() throws Excep...
        TestUtil.initWarehouse();

        // Configure the job
        job = new JobConf();

        job.setMapperClass(WordCount_Token...
        job.setCombinerClass(WordCount_Sum...
        job.setReducerClass(WordCount_SumR...

        job.setMapOutputKeySchema(SchemaUti...
        job.setMapOutputValueSchema(SchemaU...

        InputUtils.addTable(TableInfo.build...
        OutputUtils.addTable(TableInfo.buil...
    }

    @SuppressWarnings("deprecation")
    @Test
    public void testMap() throws IOExce...
        MapUTContext mapContext = new MapUT...
        mapContext.setInputSchema(INPUT_SC...
        mapContext.setOutputSchema(OUTPUT_...

        // Prepare the test data
        Record record = mapContext.createIn...
        record.set(new Text[] {new Text(...

Package and upload

After debugging your program, package it into a JAR file and upload it to MaxCompute as a resource. For more information, see Package, upload, and register a Java program.

Run MapReduce

Run the MapReduce program on the MaxCompute client.

1. In the left-side navigation pane, click Project Explorer.

2. Right-click the project name and select Open in Console.

3. In the Console pane, run the following command to run the MapReduce job.

   For more information about commands, see JAR commands.

   jar -resources wordcount.jar -classpath D:\odps\clt\wordcount.jar com.aliyun.odps.examples.mr.WordCount wc_in wc_out;