本节以EMR 3.26.0版本为例,介绍服务组件在E-MapReduce不同集群中各节点的部署情况。

背景信息

创建E-MapReduce集群时,不同类型的实例节点(实例类型)上会部署不同的服务角色,例如,Hadoop HDFS中的NameNode部署在Master节点中。如果是大规模集群,您可通过引导操作实现自定义部署和分配,例如,独立部署ZooKeeper集群和采用独立的阿里云RDS作为Hive MetaStore存储。

查看集群服务组件具体部署信息

  1. 登录阿里云E-MapReduce控制台
  2. 在顶部菜单栏处,选择地域(Region)。
  3. 单击上方的集群管理
  4. 集群管理页面,单击相应集群所在行的详情
  5. 在左侧导航栏,单击集群服务,选择查看的集群服务。
  6. 单击部署拓扑,可查看服务组件的具体部署信息。

Hadoop集群

Hadoop集群服务组件具体部署信息
  • 必选服务
    服务名 主实例节点 核心实例节点
    HDFS
    • KMS
    • SecondaryNameNode
    • HttpFS
    • HDFS Client
    • NameNode
    • DataNode
    • HDFS Client
    YARN
    • ResourceManager
    • App Timeline Server
    • JobHistory
    • WebAppProxyServer
    • Yarn Client
    • Yarn Client
    • NodeManager
    Hive
    • Hive MetaStore
    • HiveServer2
    • Hive Client
    Hive Client
    Spark
    • Spark Client
    • SparkHistory
    • ThriftServer
    Spark Client
    Knox Knox N/A
    Zeppelin Zeppelin N/A
    Tez
    • Tomcat
    • Tez Client
    Tez Client
    Ganglia
    • Gmond
    • Httpd
    • Gmetad
    • Ganglia Client
    • Gmond
    • Ganglia Client
    Pig Pig Client Pig Client
    Sqoop Sqoop Client Sqoop Client
    Bigboot
    • Bigboot Client
    • Bigboot Monitor
    • Bigboot Client
    • Bigboot Monitor
    OpenLDAP OpenLDAP N/A
    Hue Hue N/A
    SmartData
    • Jindo Namespace Service
    • Jindo Storage Service
    • Jindo Client
    • Jindo Storage Service
    • Jindo Client
  • 可选服务
    服务名 主实例节点 核心实例节点
    Flume
    • Flume Agent
    • Flume Client
    • Flume Agent
    • Flume Client
    Livy Livy N/A
    Superset Superset N/A
    Flink
    • FlinkHistoryServer
    • Flink Client
    Flink Client
    Ranger
    • RangerPlugin
    • RangerAdmin
    • RangerUserSync
    RangerPlugin
    Storm
    • Storm Client
    • UI
    • Nimbus
    • Logviewer
    • Storm Client
    • Supervisor
    • Logviewer
    Phoenix Phoenix Client Phoenix Client
    Analytics Zoo
    • Analytics-Zoo-Scala
    • Analytics-Zoo-Python
    • Analytics-Zoo-Scala
    • Analytics-Zoo-Python
    Kudu
    • Kudu Master
    • Kudu Client
    • Kudu Tserver
    • Kudu Master
    • Kudu Client
    HBase
    • HMaster
    • HBase Client
    • ThriftServer
    • HBase Client
    • HRegionServer
    ZooKeeper
    • ZooKeeper
    • ZooKeeper Client
    • ZooKeeper leader
    • ZooKeeper follower
    • ZooKeeper Client
    Oozie Oozie N/A
    Presto
    • Presto Client
    • PrestoMaster
    • Presto Client
    • PrestoWorker
    Impala
    • Impala Runtime and Shell
    • Impala Catalog Server
    • Impala StateStore Server
    • Impala Runtime and Shell
    • Impala Daemon Server

Druid集群

Druid集群服务组件具体部署信息
  • 必选服务
    服务名 主实例节点 核心实例节点
    Druid
    • Druid Client
    • Coordinator
    • Overlord
    • Broker
    • Router
    • MiddleManager
    • Historical
    • Druid Client
    HDFS
    • KMS
    • SecondaryNameNode
    • HttpFS
    • HDFS Client
    • NameNode
    • DataNode
    • HDFS Client
    Ganglia
    • Gmond
    • Httpd
    • Gmetad
    • Ganglia Client
    • Gmond
    • Ganglia Client
    ZooKeeper
    • ZooKeeper follower
    • ZooKeeper Client
    • ZooKeeper leader
    • ZooKeeper follower
    • ZooKeeper Client
    OpenLDAP OpenLDAP N/A
    Bigboot
    • Bigboot Client
    • Bigboot Monitor
    • Bigboot Client
    • Bigboot Monitor
    SmartData
    • Jindo Namespace Service
    • Jindo Storage Service
    • Jindo Client
    • Jindo Storage Service
    • Jindo Client
  • 可选服务
    服务名 主实例节点 核心实例节点
    YARN
    • ResourceManager
    • App Timeline Server
    • JobHistory
    • WebAppProxyServer
    • Yarn Client
    • Yarn Client
    • NodeManager
    Superset Superset N/A

Kafka集群

Kafka集群服务组件具体部署信息
  • 必选服务
    服务名 主实例节点 核心实例节点
    Kafka-Manager Kafka Manager N/A
    Kafka
    • Kafka Client
    • KafkaMetadataMonitor
    • Kafka Rest Proxy
    • Kafka Broker broker
    • Kafka Schema Registry
    • Kafka Broker broker
    • Kafka Client
    Ganglia
    • Gmond
    • Httpd
    • Gmetad
    • Ganglia Client
    • Gmond
    • Ganglia Client
    ZooKeeper
    • ZooKeeper follower
    • ZooKeeper Client
    • ZooKeeper leader
    • ZooKeeper follower
    • ZooKeeper Client
    OpenLDAP OpenLDAP N/A
  • 可选服务
    服务名 主实例节点 核心实例节点
    Ranger
    • RangerPlugin
    • RangerUserSync
    • RangerAdmin
    RangerPlugin
    Knox Knox N/A

ZooKeeper集群

ZooKeeper集群服务组件具体部署信息
服务名 主实例节点 核心实例节点
Ganglia N/A
  • Gmond
  • Httpd
  • Gmetad
  • Ganglia Client
ZooKeeper N/A
  • ZooKeeper leader
  • ZooKeeper Client
  • ZooKeeper follower

Flink集群

Flink集群服务组件具体部署信息

必选服务

服务名 主实例节点 核心实例节点
HDFS
  • KMS
  • SecondaryNameNode
  • HttpFS
  • HDFS Client
  • NameNode
  • DataNode
  • HDFS Client
YARN
  • ResourceManager
  • App Timeline Server
  • JobHistory
  • WebAppProxyServer
  • Yarn Client
  • Yarn Client
  • NodeManager
Ganglia
  • Gmond
  • Httpd
  • Gmetad
  • Ganglia Client
  • Gmond
  • Ganglia Client
ZooKeeper
  • ZooKeeper follower
  • ZooKeeper Client
  • ZooKeeper leader
  • ZooKeeper follower
  • ZooKeeper Client
Zeppelin Zeppelin N/A
Flink
  • FlinkHistoryServer
  • Flink Client
Flink Client
Has HASServer N/A
SmartData
  • JindoFS Namespace Service
  • JindoFS Storage Service
  • SmartData Client
  • JindoFS Storage Service
  • SmartData Client
Bigboot
  • Bigboot Client
  • Bigboot Monitor
  • Bigboot Client
  • Bigboot Monitor
OpenLDAP OpenLDAP N/A
GRAFANA GrafanaServer N/A
Prometheus PrometheusServer N/A
AlertManager AlertManager N/A