全部产品
存储与CDN 数据库 安全 应用服务 数加·人工智能 数加·大数据基础服务 互联网中间件 视频服务 开发者工具 解决方案 物联网
E-MapReduce

搭建集群提交Gateway

更新时间:2017-07-10 15:24:09

Gateway

一些客户需要自主搭建Gateway向E-MapReduce集群提交作业,目前E-MapReduce在产品页面上不支持购买Gateway,后续可以在产品上直接购买Gateway,并把Hadoop环境准备好供用户使用。

网络

首先要保证Gateway机器在EMR对应集群的安全组中,Gateway节点可以顺利的访问EMR集群。设置机器的安全组请参考ECS的安全组设置说明。

环境

  • Java环境

安装至少JDK 1.7及以上

  • 复制所有的依赖的hadoop的包到gateway
  1. scp -r root@masterip:/opt/apps/extra-jars /opt/apps/
  2. scp -r root@masterip:/usr/lib/hadoop-current /opt/apps/
  3. scp -r root@masterip:/usr/lib/hive-current /opt/apps/
  4. scp -r root@masterip:/usr/lib/spark-current /opt/apps/
  5. ln -s /opt/apps/hadoop-current /usr/lib/hadoop-current
  6. ln -s /opt/apps/hive-current /usr/lib/hive-current
  7. ln -s /opt/apps/spark-current /usr/lib/spark-current
  • 复制配置文件到gateway

EMR-3.2.0 及以上版本

  1. mkdir /etc/ecm
  2. scp -r root@masterip:/etc/ecm/hadoop-conf /etc/ecm/hadoop-conf
  3. scp -r root@masterip:/etc/ecm/hive-conf /etc/ecm/hive-conf/

EMR-3.2.0 以下版本

  1. mkdir /etc/emr
  2. scp -r root@masterip:/etc/emr/hadoop-conf /etc/emr/hadoop-conf
  3. scp -r root@masterip:/etc/emr/hive-conf /etc/emr/hive-conf/
  • 复制环境变量到gateway,并执行

    1. scp root@masterip:/etc/profile.d/hadoop.sh /etc/profile.d/
    2. source /etc/profile.d/hadoop.sh
  • 修改hosts配置将集群的master节点中的host内容复制到gateway的/etc/hosts

    1. #start add cluster host of cluster 22663,Mon May 30 19:21:51 CST 2016
    2. xx.yy.zz.tt emr-header-1.cluster-1212 emr-header-1 xxxxxxxx1
    3. xx.yy.zz.tt1 emr-worker-2.cluster-1212 emr-worker-2 emr-header-3 xxxxxxx2
    4. xx.yy.zz.tt2 emr-worker-1.cluster-1212 emr-worker-1 emr-header-2 xxxxxxxx3
    5. #end add cluster host

完成以上以后,配置就完成了。

测试

  • Hive

    1. [hadoop@iZ23bc05hrvZ ~]$ hive
    2. hive> show databases;
    3. OK
    4. default
    5. Time taken: 1.124 seconds, Fetched: 1 row(s)
    6. hive> create database school;
    7. OK
    8. Time taken: 0.362 seconds
    9. hive>
  • 运行Hadoop作业

    1. [hadoop@iZ23bc05hrvZ ~]$ hadoop jar /usr/lib/hadoop-current/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 10 10
    2. Number of Maps = 10
    3. Samples per Map = 10
    4. Wrote input for Map #0
    5. Wrote input for Map #1
    6. Wrote input for Map #2
    7. Wrote input for Map #3
    8. Wrote input for Map #4
    9. Wrote input for Map #5
    10. Wrote input for Map #6
    11. Wrote input for Map #7
    12. Wrote input for Map #8
    13. Wrote input for Map #9
    14. File Input Format Counters
    15. Bytes Read=1180
    16. File Output Format Counters
    17. Bytes Written=97
    18. Job Finished in 29.798 seconds
    19. Estimated value of Pi is 3.20000000000000000000
本文导读目录