Use Apache HBase with Apsara File Storage for HDFS

Updated at:

This topic explains how to use Apache HBase with Apsara File Storage for HDFS.

Prerequisites

  • You have activated Apsara File Storage for HDFS and created a file system instance and a mount target. For more information, see Quick start for Apsara File Storage for HDFS.

  • You have installed JDK 1.8 or later on all nodes in your Hadoop cluster.

  • You have deployed a distributed Apache HBase cluster. The HBase version must be compatible with your Hadoop version. For more information, see the Apache HBase Reference Guide.

    The examples in this topic use Hadoop-2.10.1 and HBase-2.3.7. For more information, see HBase Hadoop version compatibility.

Note

If you use the built-in HBase in Alibaba Cloud E-MapReduce, see Use E-MapReduce on File Storage HDFS.

Step 1: Mount an Apsara File Storage for HDFS instance on a Hadoop cluster

Mount the Apsara File Storage for HDFS instance on your Hadoop cluster. For more information, see Mount an Apsara File Storage for HDFS file system.

Step 2: Configure Apache HBase

  1. Copy the core-site.xml file from your Hadoop configuration to the ${HBASE_HOME}/conf directory.

    cp ${HADOOP_HOME}/etc/hadoop/core-site.xml ${HBASE_HOME}/conf
  2. Configure the ${HBASE_HOME}/conf/hbase-site.xml file.

    <property>
        <!-- Run HBase in distributed mode. -->
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <!-- Specify the HBase storage directory. Replace f-xxxxxxxxxxxxxxx.cn-xxxxxxx.dfs.aliyuncs.com with your mount target domain name. -->
        <name>hbase.rootdir</name>
        <value>dfs://f-xxxxxxxxxxxxxxx.cn-xxxxxxx.dfs.aliyuncs.com:10290/hbase</value>
    </property>
    <property>
        <!-- This property must be set to false when you use HBase with Apsara File Storage for HDFS. -->
        <name>hbase.unsafe.stream.capability.enforce</name>
        <value>false</value>
    </property>
    <property>
        <!-- ZooKeeper configuration. Modify the value based on your actual environment. -->
        <name>hbase.zookeeper.quorum</name>
        <value>hostname:2181</value>
    </property>
  3. Configure the ${HBASE_HOME}/conf/hbase-env.sh file.

    ## HBase does not use its own ZooKeeper
    export HBASE_MANAGES_ZK=false
    ## Add the Apsara File Storage for HDFS Java SDK to HBASE_CLASSPATH
    export HBASE_CLASSPATH=/path/to/aliyun-sdk-dfs-x.y.z.jar:${HBASE_CLASSPATH}

Step 3: Verify Apache HBase

  1. To start HBase, run the following command:

    ${HBASE_HOME}/bin/start-hbase.sh
    Important

    Before you start the distributed HBase cluster, ensure the ZooKeeper service is running.

  2. Create a test file, such as dfs_test.txt, with the following content.

    create 'dfs_test', 'cf'
    for i in Array(0..9999)
     put 'dfs_test', 'row'+i.to_s , 'cf:a', 'value'+i.to_s
    end
    list 'dfs_test'
    scan 'dfs_test', {LIMIT => 10, STARTROW => 'row1'}
    get 'dfs_test', 'row1'
  3. Run the following command to create a table in HBase and write 10,000 rows of data.

    ${HBASE_HOME}/bin/hbase shell dfs_test.txt

    In the command, replace dfs_test.txt with the name of the test file you created.

    ROW                                     COLUMN+CELL
     row1                                   column=cf:a, timestamp=2021-12-01T15:02:43.900, value=value1
     row10                                  column=cf:a, timestamp=2021-12-01T15:02:43.926, value=value10
     row100                                 column=cf:a, timestamp=2021-12-01T15:02:44.592, value=value100
     row1000                                column=cf:a, timestamp=2021-12-01T15:02:52.516, value=value1000
     row1001                                column=cf:a, timestamp=2021-12-01T15:02:52.517, value=value1001
     row1002                                column=cf:a, timestamp=2021-12-01T15:02:52.518, value=value1002
     row1003                                column=cf:a, timestamp=2021-12-01T15:02:52.518, value=value1003
     row1004                                column=cf:a, timestamp=2021-12-01T15:02:52.557, value=value1004
     row1005                                column=cf:a, timestamp=2021-12-01T15:02:52.558, value=value1005
     row1006                                column=cf:a, timestamp=2021-12-01T15:02:52.559, value=value1006
    10 row(s)
    Took 0.0395 seconds
    COLUMN                                  CELL
     cf:a                                   timestamp=2021-12-01T15:02:43.900, value=value1
  4. Use MapReduce to count the rows.

    Important

    Before running the MapReduce counting job on YARN, start YARN in the Hadoop cluster.

    • Run the command:

      HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase mapredcp` \
      ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/lib/hbase-mapreduce-2.3.7.jar \
      rowcounter dfs_test
    • Expected output:

      If the row count matches the number of rows written, Apache HBase is configured correctly.

      HBaseCounters
              BYTES_IN_REMOTE_RESULTS=0
              BYTES_IN_RESULTS=427780
              MILLIS_BETWEEN_NEXTS=593
              NOT_SERVING_REGION_EXCEPTION=0
              REGIONS_SCANNED=1
              REMOTE_RPC_CALLS=0
              REMOTE_RPC_RETRIES=0
              ROWS_FILTERED=0
              ROWS_SCANNED=10000
              RPC_CALLS=2
              RPC_RETRIES=0
      org.apache.hadoop.hbase.mapreduce.RowCounter$RowCounterMapper$Counters
              ROWS=10000
      File Input Format Counters
              Bytes Read=0

    If you have configured HADOOP_HOME and HADOOP_CLASSPATH as environment variables, you can also run the following command to run the RowCounter job.

    ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.mapreduce.RowCounter dfs_test