After uploading device data to IoT Platform, enterprises often face challenges with the development, management, and cost of big data links. This topic presents a business case to illustrate how to build an enterprise IoT big data application architecture using the data service of Alibaba Cloud IoT Platform.
Background information
As IoT applications grow, enterprises mine device data to expand client applications and generate service revenue. This data also supports digital operations, risk control, and efficient administration, and provides a basis for making decisions about product evolution.
Business description
In this business case, four departments use data from IoT Platform:
The information department analyzes one year of device time series data to generate business metrics for device growth and activity.
The business department queries data from the last three days for applications that analyze device start and stop cycles.
The risk control department tracks device data records from the last six months for tasks such as incident backtracking.
The client device center displays statistical metrics for individual devices to provide value-added services, increase customer engagement, and drive sales of related products.
Application architecture
The following section compares the data architectures for the business scenarios described above.
Legacy architecture
Based on the business scenarios, the enterprise initially built the following data architecture:
① and ②: Configure data forwarding from IoT Platform to Tablestore. To support the information department, this setup must be forward compatible and store time series data for at least one year.
③: Add a data forwarding rule in IoT Platform to send data to Lindorm to support the risk control service.
④: A data API uses polling to sync scheduled statistical metrics from IoT Platform to an RDS database for client queries.
Problems
The three main challenges are:
The data architecture is complex. Multiple data links introduce stability risks and increase the difficulty and cost of management and maintenance.
The Java application that uses polling to call the data API cannot meet the performance requirements for large-scale data synchronization.
It creates unnecessary storage costs. Although the information department does not require real-time analysis of time series data, an extra 362 days of data must be stored to support other services.
New architecture
To address these common scenarios, IoT Platform launched a data integration feature for its data service. You can use the IoT Reader in DataWorks to configure an integration task that directly integrates data from IoT Platform into your enterprise data warehouse.
The new enterprise architecture is shown below. It meets the needs of all four business scenarios and includes the following key changes:
The data storage duration for data forwarded to Tablestore is configured based on requirements, with data set to expire after three days.
Data from IoT Platform is stored by configuring a DataWorks integration task. A single copy of the time series data is used by the information department for analysis. You can also configure an export task to store the data in Lindorm.
The Java application no longer polls the IoT Platform data API. Instead, data is integrated into the data warehouse and then synced to an RDS database to provide metrics.
Benefits
You can plan high-performance storage duration based on requirements and select appropriate storage for offline and real-time services. This approach greatly reduces overall data storage costs.
Storage cost formula:
High-performance storage space (1 year) × T1 CNY/TB/month - [High-performance storage space (3 days) × T1 CNY/TB/month + Offline storage space × T2 CNY/TB/month].It simplifies the integration of device time series data into the enterprise data warehouse. This makes it easier to combine device data with business dimensions for analysis, which increases the data's application value.
The data links are simplified. Four data links are merged into two: one for analysis and one for transactions. This reduces the difficulty and cost of management and maintenance.