DocsICP FilingConsole
Documentation
Home Page DataWorks User Guide Data Integration Common business scenarios and best practices More scenarios and practices Automate IoT data synchronization to the cloud

Automate IoT data synchronization to the cloud

Updated at:
Copy as MD
Product
My Favorites

You can use DataWorks Data Integration to automatically synchronize data from IoT devices to the cloud for big data analytics.

Background

The Internet of Things (IoT) uses sensors and other devices to collect information in real time. By connecting to networks, IoT enables communication among objects and between people and objects for intelligent sensing, identification, and management.

IoT, big data, and cloud computing form the third wave of information technology. IoT connects things, big data extracts value from data, and cloud computing provides the computing resources that support both.

Big data analytics is a crucial part of the IoT ecosystem. The IoT architecture includes devices, networks, platforms, analytics, applications, and security. The first step in extracting value from IoT data is to move it to the cloud.

Solution

This solution involves two main parts: storing raw data and synchronizing it to an analytics system.

Large volumes of data from IoT devices are typically stored in a semi-structured format. For example, you can use OSS to store raw data as CSV files.

To synchronize this data to a big data system or a traditional database, you need a dedicated data synchronization service. The following diagram shows how DataWorks Data Integration synchronizes data from OSS to a big data system.Solution overview

  1. Create a batch sync node. For more information, see Configure in wizard mode.

  2. Select OSS as the data source for reading. For more information, see OSS Reader.

  3. Configure a destination for the data. This topic uses MaxCompute Writer as an example. You can also write data to other types of data stores. For more information, see Supported data source types, Reader plug-ins, and Writer plug-ins.

Configure an automated workflow

When you read CSV files from OSS, you must specify the filename (object prefix). IoT devices often generate CSV files continuously, making manual synchronization impractical. The following procedure shows how to automatically synchronize data to MaxCompute when a new CSV file is generated every five minutes.

This solution has the following requirements:

  • Files in OSS must be generated periodically.

    DataWorks can run tasks on a recurring schedule. You can set the schedule for the DataWorks data synchronization task to match the file generation cycle in OSS. For example, if a new file is generated in OSS every 15 minutes, configure the DataWorks task to run every 15 minutes.

  • Generated filenames must include a timestamp.

    When reading files, the OSS synchronization task requires filenames that include timestamps. DataWorks uses scheduling parameters to dynamically generate filenames that match the filenames in OSS.

    Note

    We recommend that you use a timestamp such as yyyymmddhhmm as part of the filename. For example, iot_log_201911062315.csv.

  1. Log on to the DataWorks console. In the target region, click Data Integration > Data Integration in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Integration.

  2. Add an OSS data source and a MaxCompute data source. For more information, see Add an OSS data source and Add a MaxCompute data source.

  3. Click the image.png icon in the upper-left corner of the page, and choose All Products > Data Development and O&M > DataStudio. On the Data Studio page, create a workflow. For more information, see Create a workflow.

  4. Create a batch sync node. For more information, see Create a batch sync node.

  5. On the editing page of the batch sync node, select a data source and use a scheduling parameter as the filename.

    For example, define the object prefix as iot_log_${filename}.csv. The timestamp part of the filename is used as a variable, which is replaced by a parameter in the ${...} format. You can customize the parameter name. In this example, the parameter name is filename.

    Click Scheduling Settings on the right side. In the Basic Properties > Parameter section, set the custom parameter to filename=$[yyyymmddhh24mi]. For more information, see Configure scheduling parameters.

    The custom variable $[yyyymmddhh24mi] represents a timestamp accurate to the minute. For example, 201911062315 (23:15 on November 6, 2019), 202005250843 (08:43 on May 25, 2020), and 201912012207 (22:07 on December 1, 2019).

  6. Under Scheduling Settings > Schedule, configure the Scheduling Cycle.

    Set Scheduling Cycle to Minute, and configure Start Time, Time interval, and End Time based on your requirements. For example, if you set the start time to 03:00, the time interval to 5 minutes, and the end time to 23:29, the task is scheduled every 5 minutes from 03:00 to 23:59.

    Important

    Make sure that the time interval is consistent with the parameter timestamp and file naming convention. For example, if files in OSS are generated every 15 minutes, set the time interval to 15 minutes as well.

  7. Submit and deploy the node. For more information, see Deploy a node.

  8. After the task is deployed, click O&M in the upper-right corner to go to the Auto Triggered Task or Auto Triggered Instances page. Check whether the generated tasks or instances meet your requirements. For more information, see View and manage scheduled tasks.

Previous：Writing data to MySQL partitioned tablesNext：Migrate self-managed MySQL on ECS to MaxCompute