AI 助理
文档备案控制台
官方文档
输入文档关键字查找
首页

元数据性能测试

更新时间：
复制为 MD 格式
产品详情
我的收藏

本文介绍元数据在create_write、open_read、rename和delete等操作上的性能测试方法。

注意事项

在进行open_read、rename、delete测试之前要确保文件存储 HDFS 版上已存在待测数据。如果不存在待测数据，请先使用create_open创建待测数据。

测试环境

配置名称

配置说明

计算VM配置

CPU核数：4核

内存：16 GB

机器数量：6台

网络带宽：1.5 Gbps

文件存储 HDFS 版配置

实例大小：10 TB

吞吐限速：1000 MB/s

软件配置

Apache Hadoop：Hadoop 2.7.6

测试工具

NNbench是Hadoop系统自带的基准测试组件，NNbench用于测试元数据操作的性能。

NNbench的jar包位于${HADOOP_HOME}/share/hadoop/mapreduce目录下，${HADOOP_HOME}为测试机器中的Hadoop 安装目录，NNbench的jar包名为hadoop-mapreduce-client-jobclient-x.x.x-tests.jar，使用方法如下。

本文所有命令均在${HADOOP_HOME}/bin目录下执行。

  • 执行命令，查看使用方式

    ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.6-tests.jar nnbench

  • 使用方式

    Usage: nnbench <options>
Options:
  -operation <可用操作是create_write open_read rename delete.必选项>
  -maps <map数量.默认值为1。非必选项>
  -reduces <reduce数量.默认值为1.非必选项>
  -blockSize <块大小（以字节为单位）.默认值为1.非必选项>
  -bytesToWrite <要写入的字节数.默认值为0.非必选项>
  -bytesPerChecksum <文件的每个校验和的字节数.默认值为1.非必选项>
  -numberOfFiles <要创建的文件数.默认值为1.非必选项>
  -baseDir <DFS存储路径.默认是/benchmark/NNBench.非必选项>
  -readFileAfterOpen <true或false.如果为true,则读取文件并报告平均读取时间.默认为false.非必选项>

create_write性能测试

  1. 使用20个Map，10个Reduce的MapReduce作业，创建1000个文件。

    ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.6-tests.jar nnbench -maps 20 -reduces 10 -numberOfFiles 1000  -operation create_write

  2. 执行如下命令查看在文件存储 HDFS 版上面创建的文件数量及大小。

    ./hadoop fs -count -q -h /benchmarks/NNBench/io_data

  3. 查看生成的统计信息。

    统计信息在当前测试目录下的NNBench_results.log文件中。

    TPS表示每秒处理的事务个数，测试结果为1987。

    ---------- NNBench ---------- :
                   Version: NameNode Benchmark 0.4
               Date & time: 2019-08-28 15:23:56,949
        Test Operation: create_write
            Start time: 2019-08-28 15:23:44,972
           Maps to run: 20
        Reduces to run: 10
    Block Size (bytes): 1
        Bytes to write: 0
    Bytes per checksum: 1
      Number of files: 200
    Replication factor: 1
Successful file operations: 4000
# maps that missed the barrier: 0
                # exceptions: 0
     TPS: Create/Write/Close: 1987
Avg exec time (ms): Create/Write/Close: 19.7415
    Avg Lat (ms): Create/Write: 7.7955
             Avg Lat (ms): Close: 11.8985
         RAW DATA: AL Total #1: 31182
         RAW DATA: AL Total #2: 47594
      RAW DATA: TPS Total (ms): 78966
RAW DATA: Longest Map Time (ms): 4025.0
          RAW DATA: Late maps: 0
     RAW DATA: # of exceptions: 0

open_read性能测试

  1. 使用20个Map、10个Reduce的MapReduce作业打开1000个文件。

    ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.6-tests.jar nnbench -maps 20 -reduces 10 -numberOfFiles 1000 -operation open_read

  2. 查看生成的统计信息。

    统计信息在当前测试目录下的NNBench_results.log文件中。

    TPS表示每秒处理的事务个数，测试结果为7174。

    ---------- NNBench ---------- :
                   Version: NameNode Benchmark 0.4
               Date & time: 2019-08-28 15:38:08,168
           Test Operation: open_read
               Start time: 2019-08-28 15:37:58,173
              Maps to run: 20
           Reduces to run: 10
       Block Size (bytes): 1
           Bytes to write: 0
       Bytes per checksum: 1
         Number of files: 1000
       Replication factor: 1
  Successful file operations: 20000
    # maps that missed the barrier: 0
                    # exceptions: 0
               TPS: Open/Read: 7147
Avg Exec time (ms): Open/Read: 2.2103
          Avg Lat (ms): Open: 1.3914
null
        RAW DATA: AL Total #1: 27828
        RAW DATA: AL Total #2: 0
    RAW DATA: TPS Total (ms): 44206
RAW DATA: Longest Map Time (ms): 2798.0
           RAW DATA: Late maps: 0
   RAW DATA: # of exceptions: 0

rename性能测试

  1. 使用20个Map，10个Reduce的MapReduce作业来重命名1000个文件。

    ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.6-tests.jar nnbench -maps 20 -reduces 10 -numberOfFiles 1000 -operation rename

  2. 查看生成的统计信息。

    统计信息在当前测试目录下的NNBench_results.log文件中。

    TPS表示每秒处理的事务个数，测试结果为3469。

    -------------- NNBench -------------- :
                   Version: NameNode Benchmark 0.4
               Date & time: 2019-08-28 15:40:54,836
            Test Operation: rename
                Start time: 2019-08-28 15:40:40,770
               Maps to run: 20
            Reduces to run: 10
        Block Size (bytes): 1
            Bytes to write: 0
        Bytes per checksum: 1
          Number of files: 1000
        Replication factor: 1
    Successful file operations: 20000
        # maps that missed the barrier: 0
                      # exceptions: 0
               TPS: Rename: 3469
    Avg Exec time (ms): Rename: 5.5941
         Avg Lat (ms): Rename: 5.56375
null
        RAW DATA: AL Total #1: 111275
        RAW DATA: AL Total #2: 0
        RAW DATA: TPS Total (ms): 111882
 RAW DATA: Longest Map Time (ms): 5765.0
              RAW DATA: Late maps: 0
    RAW DATA: # of exceptions: 0

delete性能测试

  1. 使用20个Map，10个Reduce的MapReduce作业来删除1000个文件。

    ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.6-tests.jar nnbench -maps 20 -reduces 10 -numberOfFiles 1000 -operation delete

  2. 查看生成的统计信息。

    统计信息在当前测试目录下的NNBench_results.log文件中。

    TPS表示每秒处理的事务个数，测试结果为10887。

    ----------  NNBench  ----------  :
                   Version: NameNode Benchmark 0.4
               Date & time: 2019-08-28 15:43:26,45
           Test Operation: delete
               Start time: 2019-08-28 15:43:17,123
              Maps to run: 20
           Reduces to run: 10
       Block Size (bytes): 1
           Bytes to write: 0
       Bytes per checksum: 1
          Number of files: 1000
       Replication factor: 1
  Successful file operations: 20000
     # maps that missed the barrier: 0
                    # exceptions: 0
               TPS: Delete: 10887
     Avg Exec time (ms): Delete: 1.5899
          Avg Lat (ms): Delete: 1.56395
null
        RAW DATA: AL Total #1: 31279
        RAW DATA: AL Total #2: 0
     RAW DATA: TPS Total (ms): 31798
 RAW DATA: Longest Map Time (ms): 1837.0
          RAW DATA: Late maps: 0
   RAW DATA: # of exceptions: 0
上一篇：集群吞吐性能测试下一篇：在文件存储 HDFS 版上使用Apache Spark