全部产品

多语言支持(thrift)

更新时间:2019-06-28 16:06:42

1、 产品支持

Thrift 提供多语言访问HBase的能力,支持的语言包从Thrift官网看括: C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, JavaScript, Node.js, Smalltalk, OCaml , Delphi 以及别的语言。主要流程是用户thrift Client 通过Thrift协议访问HBase的thriftserver,thriftserver请求转发给HBase的存储服务.大概架构图如下:

架构图

开通HBase thriftserver服务:

​ 管控页面点击 开通thriftserver服务化(高可用版本thriftserver) ,会得到一个 host:port的访问入口;

thriftserver

2、使用说明

2.1 用户Thrift client访问:

一般客户常见的访问方式是python及php ,这里给出php的访问方式;

2.1.1 以php走thrift访问HBase:

我们云HBase的thrift环境是0.9.0,所以建议客户客户端的版本也为 0.9.0,可以从这里下载thrift的0.9.0 版本,下载的源码包我们后面会用到,这里需要先安装thrift编译环境,对于源码安装可以参考thrift官网

通过如下命令可以看出安装thrift的版本信息;

  1. thrift --version
2.1.2. 生成thrift访问client的访问文件;

​ 我们从这里下载云HBase的Hbase.thrift文件,云HBase使用的是thrift1协议

​ 编译命令如下:

  1. thrift --gen <language> Hbase.thrift

​ 上述 language 是语言的缩写,常见的有如下:

  1. thrift --gen php Hbase.thrift
  2. thrift --gen cpp Hbase.thrift
  3. thrift --gen py Hbase.thrift

​ 执行thrift —gen php Hbase.thrift ,在目录下得到gen-php ,这个是需要的函数包文件;

  1. thrift git:(last_dev) ll
  2. total 56
  3. -rw-r--r-- 1 xuanling.gc staff 24K 3 5 15:06 Hbase.thrift
  4. drwxr-xr-x 3 xuanling.gc staff 96B 8 1 16:03 gen-php

​ 将下载到的Thrift源码文件夹下的/lib/php/lib下面的Thrift文件夹以及gen-php一起放在我们的业务逻辑代码一个src目录下面,加上我们自己的client.php的代码,目录结果如下所示:

  1. [root@xxxxxxxxxxx thrift_client]# ll
  2. total 12
  3. -rw-r--r-- 1 zookeeper games 2743 Aug 2 11:16 client.php
  4. drwxr-xr-x 3 zookeeper games 4096 Aug 2 01:22 gen-php
  5. drwxr-xr-x 12 zookeeper games 4096 Aug 2 01:22 Thrift
php访问代码编写;

​ 上述的Thrift文件夹以及gen-php文件夹,可以随自己项目以及个人风格命名,这里方便大家搞清目录结构,就保留原来风格;下面贴出php的代码,下面的程序是在HBase 建了一张表”new”:

  1. <?php
  2. ini_set('display_errors', E_ALL);
  3. $GLOBALS['THRIFT_ROOT'] = "/root/thrift_client";
  4. /* Dependencies. In the proper order. */
  5. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/Transport/TTransport.php';
  6. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/Transport/TSocket.php';
  7. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/Protocol/TProtocol.php';
  8. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/Protocol/TBinaryProtocol.php';
  9. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/Protocol/TBinaryProtocolAccelerated.php';
  10. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/Transport/TBufferedTransport.php';
  11. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/Type/TMessageType.php';
  12. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/Factory/TStringFuncFactory.php';
  13. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/StringFunc/TStringFunc.php';
  14. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/StringFunc/Core.php';
  15. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/Type/TType.php';
  16. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/Exception/TException.php';
  17. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/Exception/TTransportException.php';
  18. require_once $GLOBALS['THRIFT_ROOT'] . '/Thrift/Exception/TProtocolException.php';
  19. require_once $GLOBALS['THRIFT_ROOT'] . '/gen-php/Hbase/Types.php';
  20. require_once $GLOBALS['THRIFT_ROOT'] . '/gen-php/Hbase/Hbase.php';
  21. use Thrift\Protocol\TBinaryProtocol;
  22. use Thrift\Transport\TBufferedTransport;
  23. use Thrift\Transport\TSocket;
  24. use Hbase\HbaseClient;
  25. use Hbase\ColumnDescriptor;
  26. use Hbase\Mutation;
  27. $host='hb-bp12pt6alr1788y35-001.hbase.rds.aliyuncs.com';
  28. $port=9099;
  29. $socket = new TSocket($host, $port);
  30. $socket->setSendTimeout(10000); // 发送超时,单位毫秒
  31. $socket->setRecvTimeout(20000); // 接收超时,单位毫秒
  32. $transport = new TBufferedTransport($socket);
  33. $protocol = new TBinaryProtocol($transport);
  34. $client = new HbaseClient($protocol);
  35. $transport->open();
  36. ####列出表####
  37. echo "----list tables----\n";
  38. $tables = $client->getTableNames();
  39. foreach ($tables as $name) {
  40. var_dump($tables);
  41. }
  42. $tablename='new';
  43. ####写数据####
  44. echo "----write data----\n";
  45. $row = 'key';
  46. $value = 'value';
  47. $atrribute = array();
  48. $mutations = array(
  49. new Mutation(array(
  50. 'column' => 'info:cn1',
  51. 'value' => $value
  52. )),
  53. );
  54. try {
  55. $client->mutateRow($tablename, $row, $mutations, $atrribute);
  56. } catch (Exception $e) {
  57. var_dump($e);//这里自己打log
  58. }
  59. ###读数据####
  60. echo "---read data---\n";
  61. $result = $client->getRow($tablename, $row, $atrribute);
  62. var_dump($result);
  63. ###删数据####
  64. echo "---delete data---\n";
  65. $client->deleteAllRow($tablename, $row, $atrribute);
  66. echo "---get data---\n";
  67. $result = $client->getRow($tablename, $row, $atrribute);
  68. var_dump($result);
  69. ###扫描数据###
  70. $row = 'ID1';
  71. $value = 'v1';
  72. $mutations = array(
  73. new Mutation(array(
  74. 'column' => 'info:c1',
  75. 'value' => $value
  76. )),
  77. );
  78. try {
  79. $client->mutateRow($tablename, $row, $mutations, $atrribute);
  80. } catch (Exception $e) {
  81. var_dump($e);
  82. }
  83. $row = 'ID2';
  84. $value = 'v2';
  85. $mutations = array(
  86. new Mutation(array(
  87. 'column' => 'info:c1',
  88. 'value' => $value
  89. )),
  90. );
  91. try {
  92. $client->mutateRow($tablename, $row, $mutations, $atrribute);
  93. } catch (Exception $e) {
  94. var_dump($e);
  95. }
  96. $row = 'ID3';
  97. $value = 'v3';
  98. $mutations = array(
  99. new Mutation(array(
  100. 'column' => 'info:c1',
  101. 'value' => $value
  102. )),
  103. );
  104. try {
  105. $client->mutateRow($tablename, $row, $mutations, $atrribute);
  106. } catch (Exception $e) {
  107. var_dump($e);
  108. }
  109. echo 'prefix scan';
  110. $scan = $client->scannerOpenWithPrefix($tablename, 'ID', null, null);
  111. $nbRows = 100;
  112. $arr = $client->scannerGetList($scan, $nbRows);
  113. echo 'count of result :'.count($arr)."\n";
  114. var_dump($arr);
  115. foreach ($arr as $k => $TRowResult) {
  116. echo "\trow:$TRowResult->row\tcolumns(array):";
  117. foreach ($TRowResult->columns as $key => $value) {
  118. echo "key:$key\tvalue:$value->value\ttimestamp:$value->timestamp\n";
  119. }
  120. }
  121. echo 'range scan';
  122. $scan = $client->scannerOpenWithStop($tablename, 'ID0', 'ID2', null, null);
  123. $nbRows = 100;
  124. $arr = $client->scannerGetList($scan, $nbRows);
  125. echo 'count of result :'.count($arr)."\n";
  126. var_dump($arr);
  127. foreach ($arr as $k => $TRowResult) {
  128. echo "\trow:$TRowResult->row\tcolumns(array):";
  129. foreach ($TRowResult->columns as $key => $value) {
  130. echo "key:$key\tvalue:$value->value\ttimestamp:$value->timestamp\n";
  131. }
  132. }
  133. ###Increment 操作###
  134. echo "do increment on a new row";
  135. $row = 'ID4';
  136. try {
  137. $newCount = $client->atomicIncrement($tablename, $row, 'info:c1', 1234);
  138. } catch (Exception $e) {
  139. var_dump($e);
  140. }
  141. echo "new count $newCount\n";
  142. ###读写 long型数据到hbase###
  143. $row = 'ID5';
  144. $value = pack("J", 4567);
  145. $mutations = array(
  146. new Mutation(array(
  147. 'column' => 'info:c1',
  148. 'value' => $value
  149. )),
  150. );
  151. try {
  152. $client->mutateRow($tablename, $row, $mutations, null);
  153. } catch (Exception $e) {
  154. var_dump($e);
  155. }
  156. echo "---read data and print it as long ---\n";
  157. $result = $client->getRow($tablename, $row, null);
  158. foreach ($result[0]->columns as $key => $value) {
  159. $count = unpack("J*mycount", $value->value);
  160. var_dump($count);
  161. }
  162. ?>

​ 代码执行结果如下:

  1. [root@xxxxxxxxxxx thrift_client]# php client.php
  2. ----list tables----
  3. array(1) {
  4. [0]=>
  5. string(3) "new"
  6. }
  7. ----write data----
  8. ---read data---
  9. array(1) {
  10. [0]=>
  11. object(Hbase\TRowResult)#8 (3) {
  12. ["row"]=>
  13. string(3) "key"
  14. ["columns"]=>
  15. array(1) {
  16. ["info:cn1"]=>
  17. object(Hbase\TCell)#10 (2) {
  18. ["value"]=>
  19. string(5) "value"
  20. ["timestamp"]=>
  21. int(1533179795969)
  22. }
  23. }
  24. ["sortedColumns"]=>
  25. NULL
  26. }
  27. }
  28. ---delete data---
  29. ---get data---
  30. array(0) {
  31. }
2.2.python访问流程;

​ 此外还有常见的python的客户,对于python的话,有happybase这种python的第三方包含thrift的库,我们见过一些客户使用Happybase进行访问HBase thrift,参见文章;此外,python 有丰富的库,我们通过pip可以安装thrift,以及访问HBase的thrift库;执行流程如下,假设用户已经安装python以及pip:

  1. pip install thrift //安装thrift默认最新版本
  2. pip install hbase-thrift //安装hbase thrift接口库

​ 上面2步执行完成以后,既可以编写访问HBase的代码:

  1. import sys
  2. import time
  3. import os
  4. from thrift import Thrift
  5. from thrift.transport import TSocket, TTransport
  6. from thrift.protocol import TBinaryProtocol
  7. from thrift.protocol.TBinaryProtocol import TBinaryProtocolAccelerated
  8. from hbase import ttypes
  9. from hbase.Hbase import Client, ColumnDescriptor, Mutation
  10. def printRow(entry):
  11. print "row: " + entry.row + ", cols:",
  12. for k in sorted(entry.columns):
  13. print k + " => " + entry.columns[k].value,
  14. print
  15. transport = TSocket.TSocket('hb-bp12pt6alr1788y35-001.hbase.rds.aliyuncs.com', 9099)
  16. transport = TTransport.TBufferedTransport(transport)
  17. protocol = TBinaryProtocol.TBinaryProtocolAccelerated(transport)
  18. client = Client(protocol)
  19. transport.open()
  20. print "---list table--"
  21. print client.getTableNames()
  22. table="new"
  23. row="key"
  24. print "---write data---"
  25. mutations = [Mutation(column="info:cn1", value="value")]
  26. client.mutateRow(table, row, mutations)
  27. print "---get data----"
  28. printRow(client.getRow(table, row)[0])
  29. print "---delete data---"
  30. client.deleteAllRow(table, row)
  31. print "---end----"
  32. transport.close()

​ 对应上述的程序执行的结果如下:

  1. [root@Test ~]# python Hbase_client.py
  2. ---list table--
  3. ['new']
  4. ---write data---
  5. ---get data----
  6. row: key, cols: info:cn1 => value
  7. ---delete data---
  8. ---end----

Go语言访问HBase

下载优化后的thrift访问压缩包(也可以从https://github.com/sdming/goh 下载原始版本),解压后放到$GOPATH/src下

  1. wget http://public-hbase.oss-cn-hangzhou.aliyuncs.com/thrift/goh.tar.gz
  2. tar -xvzf goh.tar.gz
  3. mv github.com $GOPATH/src

示例代码请参考 $GOPATH/src/github.com/sdming/goh/demo/client.go 包含了DDL以及数据读写的代码示例