Use Apache HBase with Apsara File Storage for HDFS
This topic explains how to use Apache HBase with Apsara File Storage for HDFS.
Prerequisites
-
You have activated Apsara File Storage for HDFS and created a file system instance and a mount target. For more information, see Quick start for Apsara File Storage for HDFS.
-
You have installed JDK 1.8 or later on all nodes in your Hadoop cluster.
-
You have deployed a distributed Apache HBase cluster. The HBase version must be compatible with your Hadoop version. For more information, see the Apache HBase Reference Guide.
The examples in this topic use Hadoop-2.10.1 and HBase-2.3.7. For more information, see HBase Hadoop version compatibility.
If you use the built-in HBase in Alibaba Cloud E-MapReduce, see Use E-MapReduce on File Storage HDFS.
Step 1: Mount an Apsara File Storage for HDFS instance on a Hadoop cluster
Mount the Apsara File Storage for HDFS instance on your Hadoop cluster. For more information, see Mount an Apsara File Storage for HDFS file system.
Step 2: Configure Apache HBase
-
Copy the core-site.xml file from your Hadoop configuration to the ${HBASE_HOME}/conf directory.
cp ${HADOOP_HOME}/etc/hadoop/core-site.xml ${HBASE_HOME}/conf -
Configure the
${HBASE_HOME}/conf/hbase-site.xmlfile.<property> <!-- Run HBase in distributed mode. --> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <!-- Specify the HBase storage directory. Replace f-xxxxxxxxxxxxxxx.cn-xxxxxxx.dfs.aliyuncs.com with your mount target domain name. --> <name>hbase.rootdir</name> <value>dfs://f-xxxxxxxxxxxxxxx.cn-xxxxxxx.dfs.aliyuncs.com:10290/hbase</value> </property> <property> <!-- This property must be set to false when you use HBase with Apsara File Storage for HDFS. --> <name>hbase.unsafe.stream.capability.enforce</name> <value>false</value> </property> <property> <!-- ZooKeeper configuration. Modify the value based on your actual environment. --> <name>hbase.zookeeper.quorum</name> <value>hostname:2181</value> </property> -
Configure the ${HBASE_HOME}/conf/hbase-env.sh file.
## HBase does not use its own ZooKeeper export HBASE_MANAGES_ZK=false ## Add the Apsara File Storage for HDFS Java SDK to HBASE_CLASSPATH export HBASE_CLASSPATH=/path/to/aliyun-sdk-dfs-x.y.z.jar:${HBASE_CLASSPATH}
Step 3: Verify Apache HBase
-
To start HBase, run the following command:
${HBASE_HOME}/bin/start-hbase.shImportantBefore you start the distributed HBase cluster, ensure the ZooKeeper service is running.
-
Create a test file, such as dfs_test.txt, with the following content.
create 'dfs_test', 'cf' for i in Array(0..9999) put 'dfs_test', 'row'+i.to_s , 'cf:a', 'value'+i.to_s end list 'dfs_test' scan 'dfs_test', {LIMIT => 10, STARTROW => 'row1'} get 'dfs_test', 'row1' -
Run the following command to create a table in HBase and write 10,000 rows of data.
${HBASE_HOME}/bin/hbase shell dfs_test.txtIn the command, replace
dfs_test.txtwith the name of the test file you created.ROW COLUMN+CELL row1 column=cf:a, timestamp=2021-12-01T15:02:43.900, value=value1 row10 column=cf:a, timestamp=2021-12-01T15:02:43.926, value=value10 row100 column=cf:a, timestamp=2021-12-01T15:02:44.592, value=value100 row1000 column=cf:a, timestamp=2021-12-01T15:02:52.516, value=value1000 row1001 column=cf:a, timestamp=2021-12-01T15:02:52.517, value=value1001 row1002 column=cf:a, timestamp=2021-12-01T15:02:52.518, value=value1002 row1003 column=cf:a, timestamp=2021-12-01T15:02:52.518, value=value1003 row1004 column=cf:a, timestamp=2021-12-01T15:02:52.557, value=value1004 row1005 column=cf:a, timestamp=2021-12-01T15:02:52.558, value=value1005 row1006 column=cf:a, timestamp=2021-12-01T15:02:52.559, value=value1006 10 row(s) Took 0.0395 seconds COLUMN CELL cf:a timestamp=2021-12-01T15:02:43.900, value=value1 -
Use MapReduce to count the rows.
ImportantBefore running the MapReduce counting job on YARN, start YARN in the Hadoop cluster.
-
Run the command:
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase mapredcp` \ ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/lib/hbase-mapreduce-2.3.7.jar \ rowcounter dfs_test -
Expected output:
If the row count matches the number of rows written, Apache HBase is configured correctly.
HBaseCounters BYTES_IN_REMOTE_RESULTS=0 BYTES_IN_RESULTS=427780 MILLIS_BETWEEN_NEXTS=593 NOT_SERVING_REGION_EXCEPTION=0 REGIONS_SCANNED=1 REMOTE_RPC_CALLS=0 REMOTE_RPC_RETRIES=0 ROWS_FILTERED=0 ROWS_SCANNED=10000 RPC_CALLS=2 RPC_RETRIES=0 org.apache.hadoop.hbase.mapreduce.RowCounter$RowCounterMapper$Counters ROWS=10000 File Input Format Counters Bytes Read=0
If you have configured HADOOP_HOME and HADOOP_CLASSPATH as environment variables, you can also run the following command to run the RowCounter job.
${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.mapreduce.RowCounter dfs_test -