Overview
The Lindorm streaming engine is developed to provide a one-stop real-time data processing solution based on standard SQL syntaxes and commonly-used database concepts. The Lindorm streaming engine is applicable to various IoV, IoT, and Internet scenarios such as ETL, real-time anomaly detection, and real-time report statistics. This topic describes the scenarios and features of the Lindorm streaming engine.
Architecture
Lindorm The following figure shows the service architecture of the Lindorm stream engine:
Lindorm provides integrated storage, computing, and query capabilities based on its different engines. The Lindorm streaming engine integrates the storage and computing of streaming data and can efficiently process real-time data. In addition, the streaming engine can be seamlessly integrated with LindormTable and LindormTSDB in metadata, SQL syntaxes, and data links. The engines of Lindorm share the same storage to provide a unified database solution. The streaming engine can also be used together with the Lindorm Ganos and Lindorm AI to provide enterprise-level features.
Benefits
The Lindorm streaming engine focuses on cloud-native, high performance, and ease of use. You can use this engine to reduce the threshold and cost of real-time data processing for large amounts of data. This way, you and focus on business implementation rather than complex infrastructure maintenance. The Lindorm streaming engine can also be used to reduce the costs and increase the efficiency of existing systems.
Real-time processing: Supports data processing in seconds.
Ease of use
Standard SQL is supported.
The Lindorm streaming engine can be used together with other Lindorm engines to obtain, store, process, and query data in a unified manner.
Cost-effectiveness
It uses a decoupled storage and compute pattern to reduce data ingestion and storage costs.
The performance in standalone stream processing scenarios is double that of open-source solutions.
Open source compatibility
Supports connections to external data sources, such as Kafka and MySQL. Stream computing results can be written directly to a destination.
Supports external static dimension tables, such as MySQL and HBase, and join queries between stream data and dimension tables.
Compatible with Flink SQL and the DataStream API.
Cloud-native architecture that supports scaling
The storage and computing resources of the Lindorm streaming engine can be individually scaled.
You can rapidly scale resources.
Automatic load balancing is supported.
Scenarios
Anomaly detection
The Lindorm streaming engine can detect abnormal metrics or patterns for the time series data generated by running devices in real time. This way, you can quickly locate issues and provide early warnings in a timely manner. The Lindorm streaming engine supports multiple detection methods, such as condition judgment, pattern recognition, and AI inference.
The following figure shows the results of a sample detection task.
Geofencing
Geofencing is applicable to detect whether a vehicle deviates from a predetermined driving route, and whether a vehicle leaves or enters a specified area. The Lindorm streaming engine defines built-in data types and operators dedicated to geofencing, and provides caching and indexing for high-performance judgment in geofencing.
Data ETL
The Lindorm streaming engine can filter, enhance, and transform real-time data and then write the data to databases. The Lindorm streaming engine and LindormTSDB shares the same storage. You do not need to associate the metadata in the two engines.
Real-time reports
The Lindorm streaming engine can perform aggregation calculations on real-time data within the specific window.
Event-driven operations
The Lindorm streaming engine subscribes to and obtains data changes from LindormTable and LindormTSDB, and then processes the changed data.
