Example: Read MaxCompute resources with a Java UDTF

UDTF code example

The following shows the code for the Java UDTF.

package com.aliyun.odps.examples.udf;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.Iterator;
import com.aliyun.odps.udf.ExecutionContext;
import com.aliyun.odps.udf.UDFException;
import com.aliyun.odps.udf.UDTF;
import com.aliyun.odps.udf.annotation.Resolve;
/**
 * project: example_project 
 * table: wc_in2 
 * partitions: p1=2,p2=1 
 * columns: cola,colc
 */
@Resolve("string,string->string,bigint,string")
public class UDTFResource extends UDTF {
  ExecutionContext ctx;
  long fileResourceLineCount;
  long tableResource1RecordCount;
  long tableResource2RecordCount;
  @Override
  public void setup(ExecutionContext ctx) throws UDFException {
  this.ctx = ctx;
  try {
   InputStream in = ctx.readResourceFileAsStream("file_resource.txt");
   BufferedReader br = new BufferedReader(new InputStreamReader(in));
   String line;
   fileResourceLineCount = 0;
   while ((line = br.readLine()) != null) {
     fileResourceLineCount++;
   }
   br.close();
   Iterator<Object[]> iterator = ctx.readResourceTable("table_resource1").iterator();
   tableResource1RecordCount = 0;
   while (iterator.hasNext()) {
     tableResource1RecordCount++;
     iterator.next();
   }
   iterator = ctx.readResourceTable("table_resource2").iterator();
   tableResource2RecordCount = 0;
   while (iterator.hasNext()) {
     tableResource2RecordCount++;
     iterator.next();
   }
 } catch (IOException e) {
   throw new UDFException(e);
 }
}
   @Override
   public void process(Object[] args) throws UDFException {
     String a = (String) args[0];
     long b = args[1] == null ? 0 : ((String) args[1]).length();
     forward(a, b, "fileResourceLineCount=" + fileResourceLineCount + "|tableResource1RecordCount="
     + tableResource1RecordCount + "|tableResource2RecordCount=" + tableResource2RecordCount);
    }
}

The following pom.xml dependency is required for local testing.

<dependency>
    <groupId>com.aliyun.odps</groupId>
    <artifactId>odps-udf-local</artifactId>
    <version>0.48.0-public</version>
</dependency>

Procedure

Local testing

1. In MaxCompute Studio, create a UDTF Java class named UDTFResource using the code from the UDTF code example section.

2. Configure the run parameters based on the content of the warehouse resource in the MaxCompute Java module.

   Note

   • The input parameters are the values of the first and third columns of each row in the p1=2, p2=1 partition of the wc_in2 table in the local resource.

   • During execution, the code reads data from the local file_resource.txt file, the wc_in1 table (mapped to the table_resource1 resource), and the p1=2, p2=1 partition of the wc_in2 table (mapped to the table_resource2 resource).

   

3. Right-click the UDTFResource class name and select Run.

   

   

Client-side testing

1. In the upper-left corner of IDEA, click Project Explorer, and select Add Resource.

   

2. Add the file_resource.txt file based on your MaxCompute instance information.

   

3. In your MaxCompute project, create and populate the wc_in1 and wc_in2 sample data tables.

   CREATE TABLE wc_in1 (
     col1 STRING,
     col2 STRING,
     col3 STRING,
     col4 STRING
   );
   
   INSERT INTO wc_in1 VALUES 
   ('A1','A2','A3','A4'),
   ('A1','A2','A3','A4'),
   ('A1','A2','A3','A4'),
   ('A1','A2','A3','A4');
   
   CREATE TABLE wc_in2 (
       cola STRING,
       colb STRING,
       colc STRING
   ) PARTITIONED BY (p1 STRING, p2 STRING);
   
   ALTER TABLE wc_in2 ADD PARTITION  (p1='2',p2='1');
   
   INSERT INTO wc_in2 PARTITION (p1='2',p2='1') VALUES 
   ('three1','three2','three3'),
   ('three1','three2','three3'),
   ('three1','three2','three3'); 

4. Map the wc_in1 and wc_in2 tables that you created in MaxCompute to the table_resource1 and table_resource2 resources.

   Add the wc_in1 resource.

   

   Add the wc_in2 resource.

   

5. Package the UDTF into a JAR package, upload it to your MaxCompute project, and register it as a function named my_udtf. To deploy, right-click the UDTFResource class name and select Deploy to Server.... In the dialog box that appears, add the required file_resource.txt, table_resource1, and table_resource2 resources in the Extra resources section.

   

6. Click Project Explorer. Right-click your target MaxCompute project and select Open Console to start the MaxCompute client. Then, run an SQL command to call the UDTF.

   

   

   The following is a sample command:

   SELECT my_udtf("10","20") AS (a, b, fileResourceLineCount);