全部产品
存储与CDN 数据库 安全 应用服务 数加·人工智能 数加·大数据基础服务 互联网中间件 视频服务 开发者工具 解决方案 物联网
E-MapReduce

Pig 作业配置

更新时间:2017-06-07 13:26:11

E-MapReduce 中,用户申请集群的时候,默认为用户提供了 Pig 环境,用户可以直接使用 Pig 来创建和操作自己的表和数据。操作步骤如下。

  1. 用户需要提前准备好 Pig 的脚本,例如:

    1. ```shell
    2. /*
    3. * Licensed to the Apache Software Foundation (ASF) under one
    4. * or more contributor license agreements. See the NOTICE file
    5. * distributed with this work for additional information
    6. * regarding copyright ownership. The ASF licenses this file
    7. * to you under the Apache License, Version 2.0 (the
    8. * "License"); you may not use this file except in compliance
    9. * with the License. You may obtain a copy of the License at
    10. *
    11. * http://www.apache.org/licenses/LICENSE-2.0
    12. *
    13. * Unless required by applicable law or agreed to in writing, software
    14. * distributed under the License is distributed on an "AS IS" BASIS,
    15. * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    16. * See the License for the specific language governing permissions and
    17. * limitations under the License.
    18. */
    19. -- Query Phrase Popularity (Hadoop cluster)
    20. -- This script processes a search query log file from the Excite search engine and finds search phrases that occur with particular high frequency during certain times of the day.
    21. -- Register the tutorial JAR file so that the included UDFs can be called in the script.
    22. REGISTER oss://emr/checklist/jars/chengtao/pig/tutorial.jar;
    23. -- Use the PigStorage function to load the excite log file into the “raw” bag as an array of records.
    24. -- Input: (user,time,query)
    25. raw = LOAD 'oss://emr/checklist/data/chengtao/pig/excite.log.bz2' USING PigStorage('\t') AS (user, time, query);
    26. -- Call the NonURLDetector UDF to remove records if the query field is empty or a URL.
    27. clean1 = FILTER raw BY org.apache.pig.tutorial.NonURLDetector(query);
    28. -- Call the ToLower UDF to change the query field to lowercase.
    29. clean2 = FOREACH clean1 GENERATE user, time, org.apache.pig.tutorial.ToLower(query) as query;
    30. -- Because the log file only contains queries for a single day, we are only interested in the hour.
    31. -- The excite query log timestamp format is YYMMDDHHMMSS.
    32. -- Call the ExtractHour UDF to extract the hour (HH) from the time field.
    33. houred = FOREACH clean2 GENERATE user, org.apache.pig.tutorial.ExtractHour(time) as hour, query;
    34. -- Call the NGramGenerator UDF to compose the n-grams of the query.
    35. ngramed1 = FOREACH houred GENERATE user, hour, flatten(org.apache.pig.tutorial.NGramGenerator(query)) as ngram;
    36. -- Use the DISTINCT command to get the unique n-grams for all records.
    37. ngramed2 = DISTINCT ngramed1;
    38. -- Use the GROUP command to group records by n-gram and hour.
    39. hour_frequency1 = GROUP ngramed2 BY (ngram, hour);
    40. -- Use the COUNT function to get the count (occurrences) of each n-gram.
    41. hour_frequency2 = FOREACH hour_frequency1 GENERATE flatten($0), COUNT($1) as count;
    42. -- Use the GROUP command to group records by n-gram only.
    43. -- Each group now corresponds to a distinct n-gram and has the count for each hour.
    44. uniq_frequency1 = GROUP hour_frequency2 BY group::ngram;
    45. -- For each group, identify the hour in which this n-gram is used with a particularly high frequency.
    46. -- Call the ScoreGenerator UDF to calculate a "popularity" score for the n-gram.
    47. uniq_frequency2 = FOREACH uniq_frequency1 GENERATE flatten($0), flatten(org.apache.pig.tutorial.ScoreGenerator($1));
    48. -- Use the FOREACH-GENERATE command to assign names to the fields.
    49. uniq_frequency3 = FOREACH uniq_frequency2 GENERATE $1 as hour, $0 as ngram, $2 as score, $3 as count, $4 as mean;
    50. -- Use the FILTER command to move all records with a score less than or equal to 2.0.
    51. filtered_uniq_frequency = FILTER uniq_frequency3 BY score > 2.0;
    52. -- Use the ORDER command to sort the remaining records by hour and score.
    53. ordered_uniq_frequency = ORDER filtered_uniq_frequency BY hour, score;
    54. -- Use the PigStorage function to store the results.
    55. -- Output: (hour, n-gram, score, count, average_counts_among_all_hours)
    56. STORE ordered_uniq_frequency INTO 'oss://emr/checklist/data/chengtao/pig/script1-hadoop-results' USING PigStorage();
    57. ```
  2. 将该脚本保存到一个脚本文件中,例如叫 script1-hadoop-oss.pig,然后将该脚本上传到 OSS 的某个目录中(例如:oss://path/to/script1-hadoop-oss.pig)。

  3. 进入阿里云 E-MapReduce 控制台作业列表

  4. 单击该页右上角的创建作业,进入创建作业页面。

  5. 填写作业名称。

  6. 选择 Pig 作业类型,表示创建的作业是一个 Pig 作业。这种类型的作业,其后台实际上是通过以下的方式提交。

    1. pig [user provided parameters]
  7. 应用参数选项框中填入 Pig 命令后续的参数。例如,如果需要使用刚刚上传到 OSS 的 Pig 脚本,则填写如下:

    1. -x mapreduce ossref://emr/checklist/jars/chengtao/pig/script1-hadoop-oss.pig

    您也可以单击选择 OSS 路径,从 OSS 中进行浏览和选择,系统会自动补齐 OSS 上 Pig 脚本的绝对路径。请务必将 Pig 脚本的前缀修改为 ossref(单击切换资源类型),以保证 E-MapReduce 可以正确下载该文件。

  8. 选择执行失败后策略。

  9. 单击确定,Pig 作业即定义完成。

本文导读目录