Cluster throughput performance test

Precautions

Before you start the performance test, note the following:

• The throughput cannot exceed the ECS bandwidth. For example, if your ECS bandwidth is 1.5 Gbps, the maximum throughput is 187.5 MB/s.

• The throughput of Apsara File Storage for HDFS depends on the purchased storage capacity.

• TestDFSIO is a distributed task that involves task scheduling and result aggregation. As a result, the calculated average cluster throughput is typically slightly lower than the throughput limit of Apsara File Storage for HDFS.

• Before you run the sequential read and random read tests, ensure that the required test data exists on Apsara File Storage for HDFS. If no test data is available, run a sequential write test first to generate the data.

Test environment

Benchmarking tool

TestDFSIO is a benchmark test component included with Hadoop. It tests the I/O throughput of a Distributed File System (DFS).

The TestDFSIO JAR file is located in the ${HADOOP_HOME}/share/hadoop/mapreduce directory of your Hadoop installation, where ${HADOOP_HOME} is the installation directory. The JAR file is named hadoop-mapreduce-client-jobclient-x.x.x-tests.jar.

Run all commands in this topic from the ${HADOOP_HOME}/bin directory.

To view the usage instructions for TestDFSIO, run the following command:

./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.6-tests.jar TestDFSIO

Sequential write performance test

Estimated throughput: 1000 MB/s

1. Write files sequentially to Apsara File Storage for HDFS.

   This command writes 500 files of 4 GB each to Apsara File Storage for HDFS, uses a buffer size of 8 MB, and saves the statistics to /tmp/TestDFSIOwrite.log.

   ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.6-tests.jar TestDFSIO -write -nrFiles 500 -size 4GB -bufferSize 8388608 -resFile /tmp/TestDFSIOwrite.log

2. Run the following command to view the number and size of files written to Apsara File Storage for HDFS:

   ./hadoop fs -count -q -h /benchmarks/TestDFSIO/io_data

3. Run the following command to view the generated statistics file:

   cat /tmp/TestDFSIOwrite.log

   The following output shows sample test results.

   • Total MBytes processed: The total size of the files written.

   • Test exec time sec: The total time elapsed for the test.

   ----- TestDFSIO -----  : write
              Date & time: Tue Aug 27 20:48:50 CST 2019
          Number of files: 500
   Total MBytes processed: 2048000
        Throughput mb/sec: 46.95
   Average IO rate mb/sec: 47.03
    IO_rate std deviation: 1.95
       Test exec time sec: 2078.86

4. Calculate the cluster throughput.

   You can calculate the cluster throughput in one of two ways:

   • Calculate throughput from the statistics file.

     The cluster throughput is approximately Total MBytes processed / Test exec time sec = 985.16 MB/s.

   • Calculate cluster throughput from the ECS monitoring egress bandwidth.

     During the sequential write test, the egress bandwidth for each node is approximately 1.4 Gbps, 1.4 Gbps, 1.4 Gbps, 1.1 Gbps, 1.1 Gbps, and 1.45 Gbps.

     Cluster throughput = (1.4 + 1.4 + 1.4 + 1.1 + 1.1 + 1.45) * 1000 / 8 = 981.25 MB/s.

Sequential read performance test

Estimated throughput: 1000 MB/s

1. Read files sequentially from Apsara File Storage for HDFS.

   This command sequentially reads 500 files of 4 GB each from Apsara File Storage for HDFS, uses a buffer size of 8 MB, and saves the statistics to /tmp/TestDFSIOread.log.

   ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.6-tests.jar TestDFSIO -read -nrFiles 500 -size 4GB -bufferSize 8388608 -resFile /tmp/TestDFSIOread.log

2. Run the following command to view the generated statistics file:

   cat /tmp/TestDFSIOread.log

   The following output shows sample test results.

   • Total MBytes processed: The total size of the files read.

   • Test exec time sec: The total time elapsed for the test.

   ----- TestDFSIO -----  : read
              Date & time: Tue Aug 27 21:24:15 CST 2019
          Number of files: 500
   Total MBytes processed: 2048000
        Throughput mb/sec: 47.03
   Average IO rate mb/sec: 47.16
    IO_rate std deviation: 2.56
       Test exec time sec: 2091.61

3. Calculate the cluster throughput.

   You can calculate the cluster throughput in one of two ways:

   • Calculate throughput from the statistics file.

     The cluster throughput is approximately Total MBytes processed / Test exec time sec = 979.15 MB/s.

   • Calculate cluster throughput from the ECS monitoring ingress bandwidth.

     During the sequential read test, the ingress bandwidth for each node is approximately 1.4 Gbps, 1.45 Gbps, 1.1 Gbps, 1.4 Gbps, 1.45 Gbps, and 1.1 Gbps.

     Cluster throughput = (1.4 + 1.45 + 1.1 + 1.4 + 1.45 + 1.1) * 1000 / 8 = 987.5 MB/s.

Random read performance test

Estimated throughput: 1000 MB/s

1. Read files randomly from Apsara File Storage for HDFS.

   This command randomly reads 500 files of 1 GB each from Apsara File Storage for HDFS, uses a buffer size of 8 MB, and saves the statistics to /tmp/TestDFSIOrandomread.log.

   ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.6-tests.jar TestDFSIO -read -random -nrFiles 500 -size 1GB -bufferSize 8388608 -resFile /tmp/TestDFSIOrandomread.log

2. Run the following command to view the generated statistics file:

   cat /tmp/TestDFSIOrandomread.log

   The following output shows sample test results.

   • Total MBytes processed: The total size of the files read.

   • Test exec time sec: The total time elapsed for the test.

   ----- TestDFSIO -----  : random read
              Date & time: Wed Aug 28 10:15:00 CST 2019
          Number of files: 500
   Total MBytes processed: 513262.72
        Throughput mb/sec: 53.07
   Average IO rate mb/sec: 53.55
    IO_rate std deviation: 5.3
       Test exec time sec: 533.5

3. Calculate the cluster throughput.

   You can calculate the cluster throughput in one of two ways:

   • Calculate throughput from the statistics file.

     The cluster throughput is approximately Total MBytes processed / Test exec time sec = 962.07 MB/s.

   • Calculate cluster throughput from the ECS monitoring ingress bandwidth.

     During the random read test, the ingress bandwidth for each node is approximately 1.35 Gbps, 1.2 Gbps, 1.4 Gbps, 1.1 Gbps, 1.4 Gbps, and 1.35 Gbps.

     Cluster throughput = (1.35 + 1.2 + 1.4 + 1.1 + 1.4 + 1.35) * 1000 / 8 = 975 MB/s.