Low-cost archive database

Challenges:

• Increased storage costs. As the data volume grows, storage costs increase proportionally.

• Decreased query performance. When the amount of data in a single instance exceeds several terabytes, query performance drops sharply.

• High O&M complexity. Using sharding to address performance degradation caused by increased data volume significantly increases O&M and business development costs.

Requirements:

• Controllable storage costs. The storage cost for historical data should be one-tenth of the cost for online data.

• Elastic scaling. Compute and storage capabilities must scale horizontally and automatically. This eliminates the complex O&M challenges associated with sharding.

• Low cost for schema changes. The solution must support fast schema changes or dynamic schemas to address the long change times typically associated with archive databases.

• Low transformation cost and support for SQL-based access.

• Meeting real-time query needs. For scenarios such as querying consumer bills or chat records, the response time (RT) for historical data queries must be close to the RT for online data queries.

• Meeting data analytics needs. Historical data is accessed infrequently. However, some business scenarios require mining and analyzing the full data. For example, the Alipay annual bill.

The cloud-native multi-model database Lindorm (ApsaraDB for HBase Performance-enhanced Edition) provides low storage costs, simplified O&M, elastic scaling, and stable performance. It integrates seamlessly with relational databases and is an ideal choice for a real-time, low-cost archive database.

Solution architecture

• The LTS data synchronization service connects to relational databases such as MySQL. It supports both full and incremental data synchronization. It also provides enterprise-level synchronization capabilities such as multi-table migration, data change detection, and DDL awareness. This allows for simple and efficient data migration.

• Lindorm provides low-cost storage for large-scale data (CNY 0.12/GB/month), elastic scaling, and pay-as-you-go billing. Its multi-model and multi-endpoint processing capabilities meet the data storage needs of various scenarios. Lindorm also seamlessly integrates with open source analytics ecosystems such as Spark, Hive, Flink, and Presto. This meets complex data analysis needs and helps you effectively extract value from your data.

Solution advantages

Simple and easy to use

• You can visually configure data migration in minutes.

• You can use integrated full and incremental synchronization to effectively reduce usage costs.

• You can use capabilities such as multi-table migration and data transformation to easily meet requirements, including merging multiple tables and combining or changing fields.

• You can benefit from comprehensive monitoring and alerting, which ensures the stability of data synchronization.

Cost-effective

• You can use storage-optimized instances with storage costs as low as CNY 0.12/GB/month. While storage costs are optimized, the built-in buffer acceleration layer ensures high real-time query performance, making it an excellent storage choice for real-time archive databases.

• LindormTable offers significant improvements in throughput and latency, with a baseline performance 7 times that of open source HBase (see report). LindormTSDB incorporates innovative, high-performance schema designs that significantly improve its baseline performance.

• Lindorm supports intelligent hot and cold data separation. In scenarios where data access frequency decreases over time, such as monitoring, social chats, and transaction bills, Lindorm automatically identifies hot and cold data. It then separates the data and stores it on high-performance or low-cost media accordingly. The unit cost difference between the two can be as high as 10:1. This process is completely transparent to user read and write access. The access performance for hot data can even be accelerated.

• Lindorm supports automatic rotation compression. The system automatically selects a combination of compression algorithms, such as dictionary, prefix, Delta, and entropy encoding, based on the data type and characteristics. Compared to common industry algorithms, the overall compression ratio is improved by 10% to 30%.

Cloud-native elasticity

• Lindorm is based on an architecture that separates storage from compute. It supports independent elastic scaling of computing and storage resources. This approach effectively avoids resource waste.

• It also provides a Serverless service that offers on-demand elasticity and pay-as-you-go billing. Lindorm Serverless is built on multi-tenant data isolation, intelligent scheduling, and an elastic Infrastructure as a Service (IaaS) foundation. It provides enterprise-grade Service-Level Agreement (SLA) guarantees and meets the availability requirements of most business applications. This greatly reduces the O&M burden of capacity management for development and operations teams and eliminates stability threats caused by traffic fluctuations.

Multi-model and search

• Lindorm is compatible with mainstream open-standard interfaces such as HBase, Phoenix (SQL), and Cassandra (CQL). This compatibility effectively reduces migration costs. It also provides capabilities such as global secondary indexes, multi-dimensional retrieval, dynamic columns, and TTL. These capabilities meet the needs of various scenarios, such as metadata, orders, bills, profiles, social media, feed streams, and logs.

• You can activate the search engine with a single click (compatible with the open source Solr standard interface). It provides capabilities such as full-text index, aggregation, and complex multi-dimensional queries. This helps accelerate data retrieval and queries to meet real-time complex analysis needs.

Big data ecosystem

• Lindorm seamlessly integrates with open source big data ecosystem products such as Spark, Hive, Flink, and Presto. It supports multiple connection methods, such as API access and file reading. This lets you simply and efficiently meet your needs for large-scale data analysis.

Typical applications

• User transaction records are written to MySQL from an application. LTS synchronizes the data from MySQL to Lindorm in real-time. You can query MySQL for records from the last three months that have changing statuses. You can query Lindorm for historical transaction records that are older than three months. The historical transaction records are stored in storage-optimized instances, which reduces storage costs by more than 90%.

• For real-time retrieval needs with complex conditions, such as querying by any combination of time, location, amount, or transaction notes, you can use the capabilities of the Lindorm search engine. These capabilities include full-text index, aggregation, and complex multi-dimensional queries. This lets you easily meet user needs without requiring business transformation.

• You can use LTS to synchronize Lindorm bill data to offline computing platforms such as Spark or MaxCompute. You can perform calculations and analysis to generate operations reports based on business needs. You can then write the data back to Lindorm for real-time user queries.

Instructions

• Full and incremental synchronization for RDS

• Affordable and accessible: A technical analysis of the cloud-native multi-model database Lindorm