To meet the demands of Internet of Things (IoT) scenarios for storing heterogeneous data, managing high-concurrency throughput, providing cost-effective storage for massive data, and performing multi-dimensional data processing and analysis, Tablestore offers IoTstore. IoTstore is a one-stop storage solution that provides storage, query, retrieval, analysis, and synchronization for massive amounts of IoT data, such as device metadata, message data, and time series trajectories.
Background information
The rapid development of IoT technology has led to its widespread use in many fields, such as manufacturing, energy, construction, healthcare, transportation, and logistics. IoT applications help save resources, improve efficiency, ensure security, and reduce costs. This helps industries achieve their sustainable development goals.
In IoT scenarios, data can be classified into three main types based on its characteristics: device metadata, device message data, and device time series data. Each type has different storage requirements. The core storage requirements for these data types are as follows:
-
Device metadata: This includes device identifiers, properties, and status data. This data is characterized by high-frequency updates and requires high query performance for tasks such as device management, device filtering, and device status queries. Storage must support high-concurrency, low-latency data updates, multi-dimensional retrieval, geospatial retrieval, and real-time data computing and analysis.
-
Device message data: This includes device event reports, platform control instructions, and message pushes. This data is characterized by message ordering and many message queues, requiring a message queue model for storage. Storage must also support message order preservation, many queues, and low-cost storage for massive data.
-
Device time series data: This includes data from sensors, device monitoring, and device trajectories. This data is characterized by infrequent updates and large data volumes, requiring a time series model for storage. Storage must also support high-concurrency writes, low-cost storage for massive data, and flexible query and analysis capabilities.
Tablestore provides the one-stop IoTstore solution to meet the data storage needs of IoT scenarios. IoTstore offers a unified data storage platform for different types of data in IoT scenarios, supports data ingestion from various data sources, integrates with stream and batch computing engines, and provides data visualization.
Scenarios
As a unified data storage platform for IoT, IoTstore can be used in scenarios such as the Internet of Vehicles (IoV), smart homes, and the Industrial Internet of Things (IIoT). For more information, see Scenarios.
Overall architecture
The following figure shows the overall architecture of IoTstore.
Upstream data ingestion
IoTstore supports data ingestion from various sources, such as IoT Platform, MQTT, Kafka, EMQX, and Flink real-time data streams.
Multi-model data storage
Tablestore provides powerful data engines, including the Wide Column engine, Time Series engine, and Indexing engine.
-
Wide Column engine: A distributed data table used to store and update device metadata.
-
Time Series engine: Designed for time series data, this engine provides high-compression-ratio storage and is used to store and analyze device time series data.
-
Indexing engine: Uses storage structures such as inverted indexes and spatial indexes. It supports full-text search, searches based on any combination of fields, and statistical aggregation. It is used to retrieve device metadata and time series metadata.
Tablestore provides three data storage models for different types of structured data: Wide Column, Time Series, and Timeline.
|
Model |
Description |
|
Wide table model |
This model is similar to the Bigtable and HBase models. It can be used in various scenarios, such as metadata and big data. It supports features such as data versions, TTL, auto-increment primary key columns, conditional updates, local transactions, atomic counters, and filters. For more information, see Wide Column model. |
|
TimeSeries model |
This model is designed based on the characteristics of time series data. It can be used in scenarios such as IoT device monitoring, device data collection, and machine monitoring data. It supports features such as automatic creation of time series metadata indexes and a wide range of time series query capabilities. For more information, see TimeSeries model. |
|
Message model |
This model is designed for message data scenarios. It can be used in message scenarios such as instant messaging (IM) and feed streams. It meets the requirements of message scenarios for message ordering, massive message storage, and real-time synchronization. It also supports full-text search and multi-dimensional composite queries. For more information, see Timeline model. |
Unified SQL query
SQL query provides a unified access interface for multiple data engines. You can use SQL queries to perform complex queries and efficient analysis on data in Tablestore.
Integration with computing ecosystems and visualization tools
-
IoTstore supports seamless integration with computing components such as Spark and MaxCompute.
-
IoTstore supports integration with the open source visualization and analysis platform Grafana to visualize the results of data analysis and processing in various formats.
Core advantages
IoTstore has rich data integration and processing capabilities. It also provides a low-cost, high-performance data storage solution for massive IoT data.
Low-cost storage
-
IoTstore provides automatic tiered storage for hot and cold data. Frequently accessed data is stored in the hot tier for high-performance access, while infrequently accessed data is stored in the cold tier for low-cost storage.
-
IoTstore provides high-compression-ratio storage for large-scale time series data to effectively reduce storage costs.
Unified data storage platform
IoTstore provides a unified data storage platform for device metadata, device message data, and device time series data in IoT scenarios. Device metadata is stored using the Wide Column model, device message data is stored using the Timeline model, and device time series data is stored using the Time Series model.
Serverless architecture
-
IoTstore uses a storage-compute disaggregation architecture. The storage, compute, and write layers support independent elasticity.
-
IoTstore supports up to 10 PB of data and trillions of records in a single table, with a Transaction Per Second (TPS) of tens of millions. It provides automatic load balancing and hot spot migration without requiring manual intervention.
Easy integration
-
IoTstore provides convenient upstream data links and supports data ingestion from IoT data sources such as Kafka, Alibaba Cloud IoT Platform, MQTT, and EMQX.
-
IoTstore supports seamless integration with various computing components such as Spark, MaxCompute, and Flink.
-
IoTstore supports integration with the data visualization and analysis platform Grafana to visualize analysis results.
-
IoTstore supports delivering full and incremental data to Object Storage Service (OSS) to meet the needs for lower-cost historical data storage and larger-scale offline and near-real-time analysis in data lake scenarios.
Access security
-
IoTstore supports identity verification to ensure the privacy of user data. It also supports Virtual Private Cloud (VPC) networks and HTTPS access. It provides multiple authentication and authorization mechanisms, in addition to Alibaba Cloud account and Resource Access Management (RAM) user features. Authorization granularity can be as fine as the table and API levels.
-
IoTstore uses triplicate technology and stores data replicas on different racks to ensure strong consistency for data writes. When a write operation returns a success message, the data is guaranteed to be written to all three replicas on disk. Applications can then immediately read the latest data.
-
IoTstore uses a shared storage architecture for fast detection and recovery from single points of failure and is designed for 99.99% availability.
Next steps
-
To ingest time series data, see Ingest time series data.
-
To build a vehicle trajectory data platform using a device access platform and Tablestore, see Develop with device time series data.
-
To build a vehicle metadata management platform using a device access platform and Tablestore, see Ingest device metadata.
-
To store time series data at a low cost and quickly query and analyze it, see Analytic storage.
-
To visualize data in Tablestore, see Use Grafana to visualize data.