Service architecture

Background

As information technology advances, businesses across all industries generate an increasing variety of data. This includes structured business metadata and operational data, metrics from devices or systems, and semi-structured data such as operational data, logs, images, and files. A traditional information technology (IT) architecture uses different storage and analytics technologies for different data types to meet diverse storage, query, and analysis needs, as shown in the following figure.

This approach is a typical example of technology fragmentation. It uses different databases to process different types of data. This method has several drawbacks:

• It involves many complex technical components.

• Technology selection is complex.

• It requires long data links for data access and data synchronization.

These drawbacks create significant problems for IT system development. They require highly skilled technical staff, lead to long service launch cycles, high failure rates, and high maintenance costs. Furthermore, technology fragmentation results in a disjointed technical architecture, which hinders its evolution. Eventually, the technical architecture cannot keep pace with business growth. For example, when a business needs to support high availability (HA) across zones, global synchronization, or lower storage costs, each technical component must evolve independently. This process is time-consuming, labor-intensive, and resource-intensive. As a business grows, its data types become more diverse, and the need for differentiated data processing increases, which leads to more severe data storage fragmentation.

A major challenge in current IT development is the conflict between the growing variety of business needs for diverse data types and the increasing complexity and cost of data storage architectures. This conflict will become more prominent with the widespread adoption of new technologies, such as 5G, the Internet of Things (IoT), and intelligent connected vehicles. To solve this problem, Alibaba Cloud developed Lindorm, a cloud-native multi-model database. Lindorm provides unified storage, query, and analysis for multi-model data. As shown in the following figure, Lindorm greatly simplifies the design of the data storage architecture, significantly improves system stability, and reduces development costs compared to traditional solutions.

Overall architecture

Lindorm uses an innovative cloud-native architecture that features the separation of storage and compute and multi-model sharing and integration. This architecture meets the demands for resource decoupling and elastic scaling in cloud computing. The cloud-native distributed file system LindormDFS serves as a unified storage foundation. It supports various specialized multi-model data engines, including a wide table engine, time series engine, search engine, and stream engine. Above the multi-model engines, Lindorm provides unified SQL access for cross-model federated queries. It also offers multiple open standard interfaces, such as HBase/Cassandra, OpenTSDB/InfluxDB, Kafka, and Hadoop Distributed File System (HDFS), to support the seamless migration of existing services. Finally, Lindorm Tunnel Service (LTS) manages data forwarding between engines and the real-time capture of data changes. This enables features such as data migration, real-time subscriptions, data lake dumps, data replication to data warehouses, multi-active geo-redundancy, and backup and recovery.

Distributed file system

Lindorm DFS (LDFS) is a distributed storage system designed for cloud infrastructure. It is compatible with the HDFS protocol and can run on local disks to meet the deployment needs of large customers. LDFS provides a unified, environment-independent standard interface for multi-model engines and external computing systems. The following figure shows the overall architecture.

LDFS offers multiple specifications, such as compute-optimized, medium, and storage-optimized. It supports a mix of these storage types to handle both hot and cold data in real-world scenarios. Combined with the hot and cold data separation capabilities of the multi-model engines, LDFS allows for flexible allocation of hot and cold storage space. This helps users leverage the low cost of cloud computing for their massive data.

In scenarios such as data analytics, backup and archiving, and data import, Lindorm allows external systems to directly access the underlying files of multi-model data engines through LDFS. This greatly improves data read and write efficiency. For example, you can directly generate physical files in the underlying data format in an offline computing system and import them into Lindorm to reduce the impact on online services.

Wide table engine

LindormTable is a distributed NoSQL system designed for massive semi-structured and structured data. It is suitable for scenarios such as metadata, orders, bills, profiles, social media, feed streams, and logs. It is compatible with open standard interfaces such as HBase and Cassandra. LindormTable is based on an architecture of automatic data partitioning, multi-replica partitions, and Log-Structured Merge-tree (LSM). It provides features such as global secondary indexes, multi-dimensional retrieval, dynamic columns, and Time-to-Live (TTL). It supports trillions of rows in a single table, high concurrency, millisecond-level response times, and cross-data-center disaster recovery with strong consistency. This efficiently meets the online storage and query needs for large-scale business data. The following figure shows the overall architecture.

LindormTable uses LDFS for data persistence. Table data is automatically sharded and distributed across multiple servers in a cluster. Each partition can have one to N replicas. These replicas have primary and secondary roles. The primary and secondary replicas can be deployed in different zones to ensure high availability and strong consistency for the cluster. The data synchronization and read/write modes between primary and secondary replicas vary based on the consistency model:

• Strong consistency mode. Only the primary replica serves reads and writes. Data is asynchronously replayed to the secondary replicas. If the node where the primary replica resides fails, a secondary replica is promoted to become the new primary. Before the promotion, the system ensures that data synchronization is complete and the secondary replica has all the latest data. The Master node coordinates this entire process.

• Eventual consistency mode. Both primary and secondary replicas serve reads and writes. Data is synchronized between them to ensure eventual consistency among the replicas.

The multi-replica architecture of LindormTable is designed based on the PACELC theorem. The consistency model for each data table can be set individually, providing different levels of availability and performance. In eventual consistency mode, the server-side triggers concurrent access to multiple replicas for each read and write request under certain conditions. This significantly increases the success rate of requests and reduces response glitches. This concurrent mechanism is built on an internal asynchronous access framework, and its additional resource consumption is negligible compared to launching multiple threads. The conditions that trigger concurrent access are of two main types:

• Triggered by a time limit. You can set a GlitchTimeout for each request. If a request's runtime exceeds this value without a response, a concurrent request is sent to the other N-1 replicas. The fastest response is used as the final response.

• Blacklist avoidance. The server-side internally blacklists replicas that are slow, unresponsive, or have other issues based on timeouts, errors, and health checks. This allows requests to proactively bypass nodes with software or hardware defects, ensuring the service remains as smooth as possible. For example, in a power loss scenario, there can be a delay of one to two minutes between when a node becomes unserviceable and when its network heartbeat is lost. The multi-replica coordination design of LindormTable can greatly improve service availability in such cases.

The LSM structure of LindormTable is designed for hot and cold data separation. It allows data tables to be automatically tiered into hot and cold layers within the engine while maintaining transparent queries. The underlying LStore's hybrid management of hot and cold storage significantly reduces the overall storage cost for massive data.

LindormTable provides a data model with a flexible table schema that supports various data types. Unlike traditional relational models, LindormTable not only supports predefined field types but also allows dynamic columns to be added at any time without requiring a DDL change. This adapts to the flexible and variable nature of big data. LindormTable also supports global secondary indexes and inverted indexes. The system automatically selects the most appropriate index based on the query conditions to accelerate queries that use multiple conditions. This is especially suitable for querying massive data in scenarios such as user profiling and billing.

Time series engine

LindormTSDB is a distributed time series engine designed for massive time series data. It is compatible with open standard interfaces such as OpenTSDB. Based on the characteristics and query patterns of time series data, it uses a partitioning algorithm that combines Timerange and hash. It also features an LSM architecture and file structure optimized for time series. LindormTSDB supports low-cost storage, pre-downsampling, aggregate computing, and HA disaster recovery for massive time series data. It efficiently meets the storage and processing needs for metrics and device operational data in scenarios such as IoT and monitoring. The following figure shows the overall architecture.

TSCore is the core component of the time series engine responsible for data organization. Its overall concept is similar to the LSM structure. Data is first written to a Memchunk and then flushed to disk. However, because time series data is naturally written in sequence, the specialized time series file, TSFile, is designed to be segmented by time windows. Data is layered both physically and logically by time. This greatly reduces the I/O amplification from compaction and makes TTL and hot/cold data separation highly efficient.

TSCompute is the component responsible for real-time computation of time series data. It focuses on common needs in the monitoring field, such as downsampling and transformation, timeline aggregation. It subscribes to data through Lindorm Stream and performs all computations in memory. Therefore, it is very lightweight and efficient, making it suitable for the system's predefined computing functions. For more flexible and complex analysis needs, you can still connect to systems such as Spark and Flink to support more scenarios and adapt to business changes.

Search engine

LindormSearch is a distributed search engine designed for massive data. It is compatible with the open standard Solr interface and can seamlessly serve as an index store for the wide table and time series engines to accelerate retrieval queries. Its overall architecture is consistent with the wide table engine. It is based on a structure of automatic data partitioning, multi-replica partitions, and Lucene. It provides features such as full-text search, aggregate computing, and complex multi-dimensional queries. It supports horizontal scaling, single-write, multiple-read, cross-data-center disaster recovery, and TTL. This meets the needs for efficient retrieval of massive data. The following figure shows the details.

LindormSearch uses LDFS for data persistence. Data is automatically sharded and distributed across multiple SearchServers. Each shard has multiple replicas and supports a single-write, multiple-read model to improve the efficiency of queries and aggregations. These replicas share storage, which effectively eliminates storage redundancy between them.

In the Lindorm system, LindormSearch can function as an independent model that provides a flexible document view for semi-structured and unstructured data. This is suitable for log data analysis and full-text content search. It can also serve as a transparent index store for the wide table and time series engines. In this mode, certain fields in the wide table or time series data are automatically synchronized to the search engine through internal data links. The data model and read/write access remain unified from the user's perspective. You do not need to be aware of the search engine's existence. The system internally coordinates tasks such as data association, consistency, query aggregation, and lifecycle management across engines. This approach delivers the value of multi-model integration in a simple and transparent way.

Stream engine

LindormStream is an engine for stream data processing. It provides storage and lightweight computing for stream data. It is compatible with the Kafka API and Flink SQL, helping businesses quickly build stream-based processing and applications on Lindorm.

LindormStream contains two main components: stream storage and stream computing. The integrated deployment and tight integration of these two components support high-performance, real-time processing of stream data. The stream storage component is responsible for writing and subscribing to message log data. It is compatible with the open source Kafka API, and data is persisted in the underlying LDFS, offering advantages such as high throughput, low cost, and elasticity. The stream computing component is responsible for the real-time processing of message logs. It is compatible with Flink SQL syntax, and the computation results can be synchronized to the Lindorm wide table engine, time series engine, and other engines.

Compute engine

The compute engine is a distributed computing service based on a cloud-native architecture. The compute nodes run in Alibaba Cloud Serverless Kubernetes (ASK) container service. The compute engine supports community-edition computing models and programming interfaces. It is also deeply integrated with the features of the Lindorm storage engine, fully utilizing underlying data storage characteristics and indexing capabilities to efficiently complete distributed jobs. It provides high-performance computing services for scenarios such as data production, interactive analysis, and machine learning. During Spark job execution, the compute engine provides a job management interface. You can use the Spark web UI (SparkUI) to perform comprehensive monitoring and maintenance for Spark jobs.

AI engine

The AI engine is an engine that integrates AI capabilities within the Lindorm database for one-stop intelligent analysis and processing of multi-model data, such as time series, text, images, and audio/video. This includes large language models (LLMs), text-to-image, image-to-image, and image recognition. It lets you flexibly import pre-trained models from open source model platforms, including ModelScope and HuggingFace, using SQL. You can also upload your own models for easy model deployment and inference within Lindorm.

The Lindorm AI engine uses a cloud-native architecture. It supports the elastic deployment of inference nodes of various specifications. These inference nodes support multiple machine types, including CPU and GPU, which effectively improves the performance of model inference. In addition, the inference nodes and multi-model engines share storage. This reduces data transfer costs and enables near-data inference optimization.