HBase supports multiple compression algorithms and data block encoding to reduce storage costs and improve read performance.
Compression algorithms
Alibaba Cloud HBase supports LZO, ZSTD, GZ, LZ4, SNAPPY, and NONE (no compression). The following table compares compression ratios and decompression speeds from production workloads.
|
Business type |
Uncompressed table size |
LZO (Compression ratio/Decompression speed in MB/s) |
ZSTD (Compression ratio/Decompression speed in MB/s) |
LZ4 (Compression ratio/Decompression speed in MB/s) |
|
Monitoring |
419.75 TB |
5.82/372 |
13.09/256 |
5.19/463.8 |
|
Logs |
77.26 TB |
4.11/333 |
6.0/287 |
4.16/496.1 |
|
Risk control |
147.83 TB |
4.29/297.7 |
5.93/270 |
4.19/441.38 |
|
Consumption records |
108.04 TB |
5.93/316.8 |
10.51/288.3 |
5.55/520.3 |
-
For low-latency workloads, use LZ4.
-
For throughput-oriented workloads such as monitoring or IoT, use ZSTD.
Encoding
DataBlockEncoding reduces storage by eliminating redundancy across key-value pairs. Set DATA_BLOCK_ENCODING to DIFF.
Procedure
-
Enable compression and encoding on a table.
alter 'test', {NAME => 'f', COMPRESSION => 'lz4', DATA_BLOCK_ENCODING =>'DIFF'} -
Compression and encoding take effect after a major compaction. Because this process can be time-consuming, run the compaction during off-peak hours.
major_compact 'test'
For more information about compression and encoding, see Exploring Data Compression and Encoding in Alibaba Cloud HBase.