Forward data to DataHub
Use the Data Forwarding feature of IoT Platform to forward device data to DataHub. You can then use stream computing engines such as Realtime Compute for Apache Flink and MaxCompute to process the data and generate real-time results, including charts, alerts, and statistics.
How it works
The Data Forwarding feature forwards device messages from an IoT Platform topic to a topic within a DataHub project. You can then use DataHub to clean, analyze, and archive the data. For more information, see What is DataHub?.
In the preceding figure:
-
Data source: For information about the supported topic message types, see Data formats.
-
SQL expression: Write an SQL expression to parse and process JSON data reported by devices. For the syntax of SQL expressions, see SQL expression.
NoteBinary data is forwarded directly without being parsed.
-
Data destination: The DataHub project and topic that receive device data.
Scenarios
Write data from IoT Platform to DataHub for unified management:
Deliver data to downstream systems for analysis and archiving. This helps you build a clear data stream and unlock the value of your data.
Use DataHub with Realtime Compute to clean heterogeneous data from multiple data sources into structured data in real time. This prepares the data for further analysis.
Build a real-time data warehouse with layers for raw data, real-time details, and real-time summaries.
For more information, see DataHub Scenarios.
Limitations
-
IoT Platform supports forwarding data to DataHub. For detailed information about the supported regions, see Features by region.
-
You can use both the new and previous versions of the Data Forwarding feature to forward data to DataHub. For an example of how to use the new version, see Forward data to DataHub (new version).
-
The previous version of Data Forwarding does not support cloud gateway products and devices, or products and devices in MQTT-compliant instances. To forward data for these products and devices, you must use the new version of Data Forwarding. For more information, see Data Forwarding (new version).
-
For more limitations on forwarding device messages from an IoT Platform instance, see Limitations.
Prerequisites
You have created a DataHub project and a topic to receive data. For more information about how to use DataHub, see Project operations.
ImportantFor an Enterprise instance, the DataHub project must be in the same region as the instance.
-
Create a data forwarding rule and an SQL expression to process data. For more information, see Create a data forwarding rule.
Configure a data destination
Log on to the IoT Platform console.
On the Overview page, find the instance that you want to manage and click the instance ID or instance name.
In the navigation pane on the left, choose .
Find the rule that you want to manage and click View in the Actions column. The Data Forwarding Rule page appears.
ImportantIf the new version of the data forwarding page is displayed, click Back to Old Version in the upper-right corner. Then, find the destination rule and click View.
In the Forward Data section, click Add Operation.
-
In the Add Operation dialog box, set the parameters and click OK.
Parameter
Description
Operation
Select Send data to DataHub.
Region
Select the region where your DataHub project is located.
Project
Select a DataHub project.
You can click Create Project to go to the DataHub console and create a project. For more information, see Project operations.
Topic
Select the DataHub topic that receives data.
After you select a topic, the rules engine automatically obtains the topic's schema and maps the filtered data to it.
Note-
When you map data to the schema, use
${}. Otherwise, the system stores a constant in the table. -
The schema's data types must match the data types that the rules engine processes. Otherwise, the system cannot store the data.
You can click Create Topic to go to the DataHub console and create a DataHub topic.
Role
Select the RAM role that grants IoT Platform permission to write data to DataHub.
If you have not created the required RAM role, click Create RAM Role to go to the RAM console. Then, create a RAM role and an authorization policy. For more information, see Create a RAM role.
-
Return to the Data Forwarding page. Find the rule and click Start to enable the rule.
Example
Build a device monitoring dashboard by using a big data platform
What to do next
You can view the operational logs of Device-to-Cloud Messages and Data Forwarding in the IoT Platform console. To do this, navigate to the IoT Platform Logs tab on the page for your instance. For more information, see IoT Platform logs.
You can log on to the DataHub console to view the data that is written to DataHub. For more information, see Quick Start (Synchronization example).
You can synchronize data from the topic to other Alibaba Cloud products in real time or near real time. This lets you connect data across different Alibaba Cloud products. Supported products include MaxCompute (formerly ODPS), AnalyticDB for MySQL (ADS), ApsaraDB RDS, Tablestore, Object Storage Service (OSS), Elasticsearch, and Function Compute. For more information, see Data synchronization.
References
For instructions on how to connect a device to IoT Platform and report data, see Device connection guide.
On the instance details page, you can view the available resources for message forwarding TPS to ensure that device messages can be forwarded to DataHub successfully. For more information, see View instance information and operational data. If the available resources are insufficient, upgrade the instance. For more information, see Upgrade an instance.
For a list of frequently asked questions about DataHub, see DataHub FAQ.