Use cold storage

Background

You can provision cold storage as an additional storage space when you purchase an ApsaraDB for HBase cluster. Use table creation statements to direct data to this storage tier. ApsaraDB for HBase Performance-enhanced Edition also provides automatic separation of hot and cold data within a single table. This feature automatically moves frequently accessed data to high-performance hot storage and less-frequently accessed data to low-cost cold storage to optimize costs.

Precautions

• Cold storage has low read IOPS, with a limit of 25 per node. Therefore, cold storage is suitable only for scenarios that involve infrequent queries.

• Cold storage offers write throughput comparable to ultra disk-based hot storage.

• Cold storage is not designed for high-concurrency read operations, which may cause request failures.

• If you have a large cold storage capacity, you can request a higher read IOPS limit by submitting a ticket.

• Each core node should manage no more than 30 TB of cold data on average. If you need a single core node to manage a larger volume of cold data, submit a ticket for optimization advice.

Prerequisites

Cold storage is supported only on ApsaraDB for HBase Performance-enhanced Edition V2.1.8 and later. If your cluster runs an earlier version, the platform automatically upgrades it to the latest version during activation. Ensure that your client dependencies are AliHBase-Connector 1.0.7/2.0.7 or later, and your HBase Shell is alihbase-2.0.7-bin.tar.gz or later.

Scenarios

Cold storage is ideal for scenarios such as data archiving and storing historical or other infrequently accessed data.

Enable cold storage

Method 1: When you create an ApsaraDB for HBase Performance-enhanced Edition cluster, you can enable and configure the capacity of cold storage on the purchase page. For more information, see Create a cluster.

Use cold storage

ApsaraDB for HBase Performance-enhanced Edition allows you to set storage properties at the column family level. You can set the storage policy for one, multiple, or all column families in a table to cold storage. Once configured, data for the specified column families is stored in cold storage and does not consume the HDFS space of the cluster. You can specify this policy when you create a table or modify the properties of an existing column family.

You can create tables and modify table properties by using the Java API or HBase Shell. Before you use the Java API, install the Java SDK and configure the parameters as described in Use the Java API to access a performance-enhanced cluster. Before you use HBase Shell, download and configure it as described in Use HBase Shell to access a performance-enhanced cluster.

Specify cold storage at table creation

hbase(main):001:0> create 'coldTable', {NAME => 'f', STORAGE_POLICY => 'COLD'}

 Admin admin = connection.getAdmin();
 HTableDescriptor descriptor = new HTableDescriptor(TableName.valueOf("coldTable"));
 HColumnDescriptor cf = new HColumnDescriptor("f");
 cf.setValue("STORAGE_POLICY", AliHBaseConstants.STORAGETYPE_COLD);
 descriptor.addFamily(cf);
 admin.createTable(descriptor);

Modify table properties for cold storage

For an existing table, you can modify the properties of a column family to use cold storage. If the column family already contains data, the data is moved to cold storage only after the next major compaction.

hbase(main):011:0> alter 'coldTable', {NAME=>'f', STORAGE_POLICY => 'COLD'}

Admin admin = connection.getAdmin();
TableName tableName = TableName.valueOf("coldTable");
HTableDescriptor descriptor = admin.getTableDescriptor(tableName);
HColumnDescriptor cf = descriptor.getFamily("f".getBytes());
// Set the storage policy for the column family to cold storage.
cf.setValue("STORAGE_POLICY", AliHBaseConstants.STORAGETYPE_COLD);
admin.modifyTable(tableName, descriptor);

Modify table properties for hot storage

If a column family is currently using cold storage, you can revert it to hot storage by modifying its properties. If the column family contains data, the data moves back to hot storage only after the next major compaction.

hbase(main):014:0> alter 'coldTable', {NAME=>'f', STORAGE_POLICY => 'DEFAULT'}

// Get an Admin instance from the connection.
Admin admin = connection.getAdmin();
TableName tableName = TableName.valueOf("coldTable");
HTableDescriptor descriptor = admin.getTableDescriptor(tableName);
HColumnDescriptor cf = descriptor.getFamily("f".getBytes());
// Set the storage policy to DEFAULT, which uses hot storage.
cf.setValue("STORAGE_POLICY", AliHBaseConstants.STORAGETYPE_DEFAULT);
admin.modifyTable(tableName, descriptor);

View cold storage usage

On the Cold Storage page in the console, you can view the overall usage of cold storage. You can also click Cold Storage Scaling to increase the capacity.

In the Cluster Management System, the Tables tab displays the amount of cold and hot storage used by each table.