Deploy a high-performance Spark cluster using eRDMA-Elastic Compute Service(ECS)-阿里云帮助中心

Elastic Remote Direct Memory Access (eRDMA) provides ultra-low latency for faster request processing. This topic describes how to create a Spark cluster that uses eRDMA-enhanced Elastic Compute Service (ECS) instances as nodes. You will also deploy a benchmark test to measure the performance of the Spark cluster when processing payloads.

Background information

Benchmark is a performance testing tool. It measures metrics such as payload running time, transfer speed, throughput, and resource usage.

Step 1: Prepare the environment

Before you test the cluster's performance, you must prepare the required environment. This includes setting up Hadoop and Spark machines, installing Hadoop, and configuring eRDMA.

Prepare a Hadoop environment. If you already have a big data cluster, you can skip this step.
- Hardware and software requirements
  Prepare your Hadoop and Spark machines with the following configurations.
  - Hadoop version: Hadoop 3.2.1
  - Spark version: Spark 3.2.1
  - ECS instances:
    Instance type: See Instance types
    vCPUs: 16
    Number of cluster nodes: 1 master node and 3 worker nodes
- Installation steps
  For more information about how to install a Hadoop big data cluster, see Automatically spin up a big data cluster using FastMR.
Log on to the ECS instance on the master node.
For more information, see Connect to a Linux instance using Workbench.
Configure eRDMA.
- Driver installation
  For more information, see Enable eRDMA on an enterprise-level instance.
- Network configuration
  1. Run the following command to open the hosts file.
```
vim /etc/hosts
```
  2. Press the i key to enter edit mode and make the following changes to the configuration file.
```
192.168.201.83 poc-t5m0        master1
192.168.201.84 poc-t5w0
192.168.201.86 poc-t5w1
192.168.201.85 poc-t5w2
```
    Note
    Replace the IP addresses with the actual IP addresses of the eRDMA network interface controllers (NICs).
  3. Press the Esc key to exit edit mode. Enter :wq and press the Enter key to save and exit the file.
- YARN configuration
  Note
  If the default NIC of the ECS instance supports eRDMA, you do not need to configure Yet Another Resource Negotiator (YARN).
  1. Run the following commands in sequence to open the yarn-env.sh file.
```
cd /opt/hadoop-3.2.1/etc/hadoop
vim yarn-env.sh
```
  2. Press the i key to enter edit mode and add the following content to the file:
```
RDMA_IP=`ip addr show eth1 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1`
export YARN_NODEMANAGER_OPTS="-Dyarn.nodemanager.hostname=$RDMA_IP"
```
    Note
    Replace eth1 with the actual name of the eRDMA NIC.
  3. Press the Esc key to exit edit mode. Enter :wq and press the Enter key to save and exit the file.
- Spark configuration
  Note
  If the default NIC of the instance supports eRDMA, you do not need to configure Spark.
  1. Run the following commands in sequence to open the spark-env.sh file.
```
cd /opt/spark-3.2.1-bin-hadoop3.2/conf
vim spark-env.sh
```
  2. Press the i key to enter edit mode and add the following content to the file.
```
export SPARK_LOCAL_IP=`/sbin/ip addr show eth1 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1`
```
    Note
    Replace eth1 with the actual name of the eRDMA NIC.
  3. Press the Esc key to exit edit mode. Enter :wq and press the Enter key to save and exit the file.
Run the following command to start Hadoop Distributed File System (HDFS) and YARN.
```
$HADOOP_HOME/sbin/start-all.sh
```

Step 2: Download the Benchmark installation package

This section describes how to download the Benchmark installation package for testing.

Run the following command to download the Benchmark installation package.

wget https://mracc-release.oss-cn-beijing.aliyuncs.com/erdma-spark/spark-erdma-jverbs.tar.gz

Run the following command to decompress the spark-erdma-jverbs.tar.gz installation package.
```
tar -zxvf spark-erdma-jverbs.tar.gz
```
The installation package contains the following files:
- erdmalib: The native library (libdisni.so) required to run spark-erdma.
- plugin-sparkrdma: The plugin and dependency libraries (spark-eRDMA-1.0-for-spark-3.2.1.jar and disni-2.1-jar-with-dependencies.jar) that support Spark RDMA.

Step 3: Run a Benchmark test

This section describes how to use Benchmark to test the payload-processing performance of the Spark cluster.

Run the following command to modify the IP route.
Note
If the default NIC of the instance supports eRDMA, you can skip this step.
```
route del -net 192.168.201.0 netmask 255.255.255.0 metric 0 dev eth0 && \
route add -net 192.168.201.0 netmask 255.255.255.0 metric 1000 dev eth0
```
Note
Replace the IP address with the gateway IP address of the actual eRDMA NIC.

Configure Spark.

Run the following command to open the spark-jverbs-erdma.conf configuration file.
```
vim /opt/spark-3.2.1-bin-hadoop3.2/conf/spark-jverbs-erdma.conf
```

Press the i key to enter edit mode and replace the content of the configuration file with the following content.

spark.master yarn
spark.deploy-mode client
# Driver
spark.driver.cores 4
spark.driver.memory 19g
# Executor
spark.executor.instances 12
spark.executor.memory 10g
spark.executor.cores 4
spark.executor.heartbeatInterval   60s
# Shuffle
spark.task.maxFailures 4
spark.default.parallelism 36
spark.sql.shuffle.partitions 192
spark.shuffle.compress            true
spark.shuffle.spill.compress      true

# Other
spark.network.timeout 3600
spark.sql.broadcastTimeout 3600
spark.eventLog.enabled             false
spark.eventLog.dir                 hdfs://master1:9000/sparklogs
spark.eventLog.compress            true
spark.yarn.historyServer.address   master1:18080
spark.serializer                  org.apache.spark.serializer.KryoSerializer

# eRDMA
spark.driver.extraLibraryPath   /path/erdmalib
spark.executor.extraLibraryPath   /path/erdmalib
spark.driver.extraClassPath       /path/spark-eRDMA-1.0-for-spark-3.2.1.jar:/path/disni-2.1-jar-with-dependencies.jar
spark.executor.extraClassPath     /path/spark-eRDMA-1.0-for-spark-3.2.1.jar:/path/disni-2.1-jar-with-dependencies.jar
spark.shuffle.manager org.apache.spark.shuffle.sort.RdmaShuffleManager
spark.shuffle.sort.io.plugin.class org.apache.spark.shuffle.rdma.RdmaLocalDiskShuffleDataIO
spark.shuffle.rdma.recvQueueDepth  128

Note

Set spark.shuffle.compress to false to achieve a better acceleration ratio.
The Spark resource configurations in this example, such as spark.executor.instances, spark.executor.memory, spark.executor.cores, and spark.sql.shuffle.partitions, are for an instance with 32 vCPUs and 128 GB of memory. Adjust these settings based on your actual instance or cluster size.

Press the Esc key to exit edit mode. Enter :wq and press the Enter key to save and exit the file.

Run the following commands in sequence to generate data.

cd /opt/spark-3.2.1-bin-hadoop3.2/conf
spark-submit --properties-file /opt/spark-3.2.1-bin-hadoop3.2/conf/spark-normal.conf --class com.databricks.spark.sql.perf.tpcds.TPCDS_Bench_DataGen spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar hdfs://master1:9000/tmp/tpcds_400 tpcds_400 400 parquet

Note

The value 400 represents the volume of data to generate, in GB. Adjust this value based on your actual cluster size.

Run the following command to run the benchmark test.
```
spark-submit --properties-file /opt/spark-3.2.1-bin-hadoop3.2/conf/spark-jverbs-erdma.conf --class com.databricks.spark.sql.perf.tpcds.TPCDS_Bench_RunAllQuery spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar all hdfs://master1:9000/tmp/tpcds_400 tpcds_400 /tmp/tpcds_400_result
```
The test is complete when output similar to the following is displayed. You can view the payload running time of the Spark cluster in the test results.
Note
You can use the same method to compare performance. To do so, remove the eRDMA plugin configurations from the Spark conf directory or log on to a different cluster that does not support eRDMA, and then run the test again.