Configuration guide
Choosing the right cluster specification is the first step to using CDP. The main factors to consider for your configuration are your use case, estimated data volume, service reliability, and budget.
The Basic Edition of a CDP cluster has three instance types: master node, utility node, and core node.
The Professional Edition of a CDP cluster has four instance types: master node, utility node, core node, and Data-Service node.
Master node selection
Master nodes are mainly used to deploy services such as HDFS NameNode, YARN ResourceManager, and ZooKeeper Server.
For a production cluster, deploy at least two master nodes to achieve high availability (HA). If you have more than one master node, HA is automatically enabled for HDFS, YARN, and Hive.
Master nodes mainly store HDFS metadata in memory. Because of this, choose an instance type with 16 GB of memory or more based on the number of files in HDFS.
Utility node selection
Each cluster has only one utility node. The utility node is mainly used to deploy CM (the CDP management platform), Hive Metastore, Hue, and cluster metadata. For high availability clusters, the utility node is also used to deploy the HDFS JournalNode and the ZooKeeper Server. For high-security clusters, the utility node is also used to install the Solr server, Ranger, and the Knox proxy.
The utility node plays a vital role in the cluster. It stores a large amount of cluster metadata and maintains the CDP management platform. Therefore, the availability of the utility node is critical. The utility node only supports cloud disk storage to ensure data reliability.
Core node selection
Core nodes are mainly used for data storage and computation. They run HDFS DataNode and YARN NodeManager.
A cluster requires at least three core nodes. The node selection depends mainly on the data volume of the cluster:
If the HDFS data volume is greater than 60 TB, choose a local disk instance type, such as the ecs.d1 instance family.
If the HDFS data volume is less than 60 TB, consider using ultra disks or standard SSDs.
Data-Service node selection
Data-Service nodes are mainly used for data computation. They include three services: Data Warehouse, Machine Learning, and Data Engineering (Spark). A cluster requires at least three Data-Service nodes, and each node must have a minimum configuration of 16 cores and 128 GB of memory.