Scenario: Cost-effective archive databases
Lindorm works alongside a relational database to meet requirements such as low storage cost, easy O&M, and elastic scaling. This topic describes the architecture and advantages of this solution.
LTS instances purchased after March 10, 2023 no longer support the cost-effective archive database scenario. LTS instances purchased before March 10, 2023 still support this scenario.
Background information
In the era of highly developed mobile internet, a large volume of business data is generated every day. As time passes and businesses grow, data volumes increase rapidly, while the access frequency of historical data decreases over time. Storing all of this data in a relational database creates a series of problems.
Challenges
Rising storage costs. Storage cost is proportional to data volume. Rapid data growth leads to a rapid increase in storage costs.
Degraded query performance. Once a single instance exceeds 1 TB of storage, query performance drops sharply.
High O&M complexity. Using database and table sharding to solve the performance degradation caused by growing data volume significantly increases O&M and development costs.
Requirements
Controllable storage costs. The storage cost of historical data is one-tenth that of online data.
Elastic scaling. Compute and storage capabilities scale horizontally and automatically, eliminating the need for database and table sharding to solve O&M challenges.
Low schema change cost. Supports fast schema changes or dynamic schemas, solving the issue of long schema-change windows in historical databases.
Low migration cost. Supports SQL-based access.
Real-time query support. For scenarios such as consumption bills and chat records, the query response time for historical data must be close to that of online data.
Support for data analytics. Historical data has low access frequency, but some business scenarios require mining and analysis over the full dataset—such as year-end billing summaries on Alipay.
Lindorm can meet requirements such as low storage cost, simple O&M, elastic scaling, and stable performance. Combined with a relational database, it is the optimal choice for a real-time, cost-effective archive database.
Solution architecture
The LTS data synchronization service connects to relational databases such as MySQL and supports one-stop full and incremental data synchronization. It also provides enterprise-grade capabilities such as multi-table migration, data transformation, and DDL awareness, helping users complete data migration simply and efficiently.
Lindorm provides low-cost storage for massive data (0.11 CNY/GB/month), elastic scaling with pay-as-you-go billing, and multi-model, multi-endpoint processing to meet users’ storage needs across various scenarios. It also seamlessly integrates with open source analytics ecosystems such as Spark, Hive, Flink, and Presto to meet complex analysis needs and maximize data value.
Solution advantages
Easy to use
Visual data migration configuration—complete migration setup in minutes.
One-stop full and incremental synchronization minimizes user cost.
Supports capabilities such as multi-table migration and data transformation, helping users easily complete tasks such as multi-table merging and column combination changes.
Comprehensive monitoring and alerting ensure the stability of data synchronization.
Cost-effectiveness
Supports capacity-optimized storage, with costs as low as 0.11 CNY/GB/month. While optimizing storage costs, the built-in buffer acceleration layer ensures strong query real-time performance, making it the optimal storage choice for real-time archive databases.
In terms of performance, the Lindorm wide table engine achieves major breakthroughs in throughput and latency. Its benchmark performance is 7x that of open source Apache HBase. For details, see Test results. The Lindorm time series engine incorporates many innovative high-performance structural designs, and its benchmark performance ranks first in the current China Academy of Information and Communications Technology (CAICT) list.
Supports intelligent hot/cold data separation. For scenarios where data gradually becomes cold over time—such as monitoring, social chat, and transaction bills—Lindorm automatically identifies hot and cold data and separates storage across high-performance and low-cost media (the price ratio between the two can reach 10:1). Read and write access remains completely transparent to users, and hot data access performance is further accelerated.
Supports adaptive compression. Based on data type and characteristics, the system automatically selects hybrid compression algorithms such as dictionary, prefix, delta, and entropy encoding. Compared with common industry algorithms, the overall compression ratio improves by 10% to 30%.
Cloud native elasticity
Based on a storage-compute separation architecture, Lindorm supports independent elastic scaling of compute and storage resources, avoiding resource waste.
Serverless service is also available, enabling on-demand instant elasticity and pay-per-usage billing. Lindorm Serverless is built on multi-tenant isolation, intelligent scheduling, and an elastic IaaS foundation. It provides enterprise-grade SLA guarantees, meets the availability requirements of most internal business operations, and significantly reduces capacity management overhead for frontline teams, eliminating stability risks caused by traffic fluctuations.
Multi-model & retrieval
Compatible with mainstream open source standard interfaces such as Apache HBase, Apache Phoenix (SQL), and Cassandra (CQL), minimizing user migration cost. Also supports global secondary indexes, multi-dimensional retrieval, dynamic columns, and TTL, meeting the needs of scenarios such as metadata, orders, bills, user profiles, social feeds, and logs.
One-click search engine activation (compatible with the open source Apache Solr standard interface). Provides full-text retrieval, aggregation computing, and complex multi-dimensional queries to accelerate retrieval and meet complex real-time analysis needs.
Big data ecosystem
Seamlessly integrates with open source big data ecosystem products such as Apache Spark, Apache Hive, Apache Flink, and Presto, supporting API access and file reads. This simply and efficiently meets users’ needs for massive data analysis.
Typical applications
User transaction records are written to MySQL through an app. LTS synchronizes MySQL data to Lindorm in real time. Records that have been changing within the past three months are queried from MySQL, while transaction records older than three months are queried from Lindorm. Historical transaction records are stored on capacity-optimized storage, reducing storage cost by more than 90%.
For users’ complex real-time retrieval needs—for example, any combination of timestamp, location, amount, and transaction remark content—the full-text retrieval, aggregation computing, and complex multi-dimensional query capabilities of the Lindorm search engine can be used. User needs can be met without any application modification.
Through LTS, Lindorm billing data can be synchronized to offline computing platforms such as Spark or MaxCompute for computation. Reports are generated based on business needs and then written back to Lindorm for real-time queries.