本文介绍如何配置Pig类型的作业。

前提条件

  • 已创建好项目,详情请参见项目管理
  • 已准备好Pig的脚本,示例如下。
     /*
     * Licensed to the Apache Software Foundation (ASF) under one
     * or more contributor license agreements.  See the NOTICE file
     * distributed with this work for additional information
     * regarding copyright ownership.  The ASF licenses this file
     * to you under the Apache License, Version 2.0 (the
     * "License"); you may not use this file except in compliance
     * with the License.  You may obtain a copy of the License at
     *
     *     http://www.apache.org/licenses/LICENSE-2.0
     *
     * Unless required by applicable law or agreed to in writing, software
     * distributed under the License is distributed on an "AS IS" BASIS,
     * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     * See the License for the specific language governing permissions and
     * limitations under the License.
     */
     -- Query Phrase Popularity (Hadoop cluster)
     -- This script processes a search query log file from the Excite search engine and finds search phrases that occur with particular high frequency during certain times of the day. 
     -- Register the tutorial JAR file so that the included UDFs can be called in the script.
     REGISTER oss://emr/checklist/jars/chengtao/pig/tutorial.jar;
     -- Use the  PigStorage function to load the excite log file into the “raw” bag as an array of records.
     -- Input: (user,time,query) 
     raw = LOAD 'oss://emr/checklist/data/chengtao/pig/excite.log.bz2' USING PigStorage('\t') AS (user, time, query);
     -- Call the NonURLDetector UDF to remove records if the query field is empty or a URL. 
     clean1 = FILTER raw BY org.apache.pig.tutorial.NonURLDetector(query);
     -- Call the ToLower UDF to change the query field to lowercase. 
     clean2 = FOREACH clean1 GENERATE user, time, org.apache.pig.tutorial.ToLower(query) as query;
     -- Because the log file only contains queries for a single day, we are only interested in the hour.
     -- The excite query log timestamp format is YYMMDDHHMMSS.
     -- Call the ExtractHour UDF to extract the hour (HH) from the time field.
     houred = FOREACH clean2 GENERATE user, org.apache.pig.tutorial.ExtractHour(time) as hour, query;
     -- Call the NGramGenerator UDF to compose the n-grams of the query.
     ngramed1 = FOREACH houred GENERATE user, hour, flatten(org.apache.pig.tutorial.NGramGenerator(query)) as ngram;
     -- Use the  DISTINCT command to get the unique n-grams for all records.
     ngramed2 = DISTINCT ngramed1;
     -- Use the  GROUP command to group records by n-gram and hour. 
     hour_frequency1 = GROUP ngramed2 BY (ngram, hour);
     -- Use the  COUNT function to get the count (occurrences) of each n-gram. 
     hour_frequency2 = FOREACH hour_frequency1 GENERATE flatten($0), COUNT($1) as count;
     -- Use the  GROUP command to group records by n-gram only. 
     -- Each group now corresponds to a distinct n-gram and has the count for each hour.
     uniq_frequency1 = GROUP hour_frequency2 BY group::ngram;
     -- For each group, identify the hour in which this n-gram is used with a particularly high frequency.
     -- Call the ScoreGenerator UDF to calculate a "popularity" score for the n-gram.
     uniq_frequency2 = FOREACH uniq_frequency1 GENERATE flatten($0), flatten(org.apache.pig.tutorial.ScoreGenerator($1));
     -- Use the  FOREACH-GENERATE command to assign names to the fields. 
     uniq_frequency3 = FOREACH uniq_frequency2 GENERATE $1 as hour, $0 as ngram, $2 as score, $3 as count, $4 as mean;
     -- Use the  FILTER command to move all records with a score less than or equal to 2.0.
     filtered_uniq_frequency = FILTER uniq_frequency3 BY score > 2.0;
     -- Use the  ORDER command to sort the remaining records by hour and score. 
     ordered_uniq_frequency = ORDER filtered_uniq_frequency BY hour, score;
     -- Use the  PigStorage function to store the results. 
     -- Output: (hour, n-gram, score, count, average_counts_among_all_hours)
     STORE ordered_uniq_frequency INTO 'oss://emr/checklist/data/chengtao/pig/script1-hadoop-results' USING PigStorage();
  • 已保存该脚本文件script1-hadoop-oss.pig,并上传到OSS的某个目录中,例如oss://path/to/script1-hadoop-oss.pig

操作步骤

  1. 进入数据开发的项目列表页面。
    1. 通过阿里云账号登录阿里云E-MapReduce控制台
    2. 在顶部菜单栏处,根据实际情况选择地域和资源组
    3. 单击上方的数据开发页签。
  2. 单击待编辑项目所在行的作业编辑
  3. 新建Pig类型作业。
    1. 在页面左侧,在需要操作的文件夹上单击右键,选择新建作业
    2. 新建作业对话框中,输入作业名称作业描述,从作业类型下拉列表中选择Pig作业类型。
      表示创建的作业是一个Pig作业。这种类型的作业,实际是通过以下方式提交的Pig作业运行。
      pig [user provided parameters]
    3. 单击确定
  4. 编辑作业内容。
    1. 作业内容中,填写提交该作业需要提供的命令行参数。
      例如,如果需要使用刚刚上传到OSS的Pig脚本,则填写的内容如下。
      -x mapreduce ossref://emr/checklist/jars/chengtao/pig/script1-hadoop-oss.pig
      说明 您也可以单击下方的+插入OSS路径,选择文件前缀OSSREF,从文件路径中进行浏览和选择,系统会自动补齐OSS上Pig脚本的路径。
    2. 单击保存,作业内容编辑完成。