Forward data to a Tablestore TimeSeries model

How it works

The Data Forwarding feature forwards device topic messages to a data table in a Tablestore instance by using the wide-column engine of Tablestore.

In the preceding figure:

• Data source: For information about supported topic message types, see Data formats (for devices that are not connected to a cloud gateway), Custom topics (MQTT cloud gateways), Message forwarding topics (GB/T 32960 cloud gateways), Message forwarding topics (JT/T 808 cloud gateways), and Message forwarding topics (SL 651 cloud gateways).

• Data destination: A data table in the created Tablestore instance that is used to receive device data.

• Parser script: Uses the writeTableStoreTs(destinationId, timeInUs, metricName, tags, fields, dataSource) function to forward topic data to a Tablestore data table.

  The data parameter supports only the Map data type, where keys correspond to column names in the table and values correspond to column values. For more information about the function, see Function list.

Limitations

• The new version of the Data Forwarding feature supports forwarding data to a Tablestore TimeSeries model only for Enterprise Edition instances in the China (Shanghai), China (Beijing), China (Shenzhen), Singapore, and Japan (Tokyo) regions.

• Your IoT Platform instance must be in a region that supports forwarding data to a Tablestore TimeSeries model. For more information about supported regions, see Feature descriptions by region.

• For more limitations on forwarding device messages from an IoT Platform instance, see Limitations.

Create a data destination

1. Log on to the IoT Platform console.

2. On the Overview page, find your instance and click the instance name to go to the Instance Details page.

3. In the navigation pane on the left, choose Message Forwarding > Data Forwarding.

4. On the Data Forwarding page, click Try New Version in the upper-right corner to open the new feature page.

   Note

   If you have already performed this operation, you will be taken directly to the new feature page the next time you access the Data Forwarding page.

5. On the Data Destinations tab, click Create Data Destination.

6. In the Create Data Destination dialog box, enter a name for the data destination, such as DataPurpose. Configure the parameters as described in the following table, and then click OK.

   Note

   Only data in the JSON format can be forwarded.

   Parameter	Description
   Select operation	Select Store Data to Tablestore.
   Region	Select the region where your Tablestore instance is located.
   Instance	Select a Tablestore instance. You can click Create Instance to navigate to the Tablestore console and create a TimeSeries model instance. For more information, see Create an instance of the TimeSeries model.
   Storage type	Select TimeSeries Model.
   Data table	Select the Tablestore time series table that will receive the data. You can click Create Data Table to navigate to the Tablestore console and create a time series table. For more information, see Create a time series table.
   Role	Grant IoT Platform the permissions to write data to the Tablestore time series table. If you have not created the required RAM role, click Create RAM Role to go to the RAM console. Then, create a RAM role and an authorization policy. For more information, see Create a RAM role.

Configure and start a parser

1. Create a parser, such as DataParser. For more information, see Step 1: Create a parser.

2. On the Resolver Details page, associate the data source.

   1. In the configuration wizard, under Data Source, click Associate Data Source.

   2. In the dialog box, select DataSource from the Data Source drop-down list and click OK.

3. On the Resolver Details page, you can associate data destinations.

   1. In the configuration wizard, click Data Destination. Then, in the upper-right corner of the Data Destination list, click Associate Data Destination.

   2. In the dialog box that appears, from the Data Destination drop-down list, select DataPurpose and click OK.

   3. In the data destination list, find the Data Destination ID and record it. For example, 1000.

      You will need this Data Destination ID in the parsing script.

4. On the parser details page, click the Parser Script tab.

5. In the script editor, enter a parser script.

   Parser scripts are similar to JavaScript. For more information about the script syntax, see Script syntax.

   For more information about function parameters, see Function list.

   // Use the payload() function to get the message content reported by the device and convert it to the JSON format.
   var data = payload("json");
   // Get the reported property values.
   var h = data.items.Humidity.value;
   var t = data.items.Temperature.value;
   
   var datasource = deviceName();
   var fields = {"temperature":t, "humidity":h};
   var tags = {};
   tags.put("region", "cn-shanghai");
   var metricName = "property";
   var timeInUs = timestamp()*1000;
   
   // In the writeTableStoreTS method, write the temperature and humidity values to the corresponding columns based on the timeline.
   writeTableStoreTs(1000, timeInUs, metricName, tags, fields, datasource);

   You can also set temperature and humidity as separate metric names and write the data by using the following expression:

   writeTableStoreTs(1000, [
       {
           "timeInUs": timestamp() * 1000,
           "metricName": "Temperature",
           "tags":tags,
           "fields":{"temperature": t},
           "dataSource": deviceName()
       },
       {
           "timeInUs": timestamp() * 1000,
           "metricName": "Humidity",
           "tags":tags,
           "fields":{"humidity": h},
           "dataSource": deviceName()
       },
   ]);

6. Click Debugging. Follow the on-screen instructions to select a product and a device, enter a topic and payload data, and verify that the script can run.

   The Topic ends with thing/event/property/post. In the JSON of the Payload data, items includes reported temperature and humidity properties such as Humidity (value: 38) and Temperature (value: 25).

   The following output indicates that the script runs as expected.

   action:
     transmit to ots[destinationId=1000], data:[{"data":{xxx
   variables:
     metricName : property
     data : {"deviceType":"Ammeter","iotId":"Exxx"}
     t : 25
     datasource : Exxx:q
     h : 38
     timeInUs : 1658738826768000
     fields : {"temperature":25,"humidity":38}
     tags : {"region":"cn-shanghai"}

7. Click Publish.

8. Return to the Resolver tab on the Data Forwarding page and click Start for the DataParser resolver.

Next steps

1. In the IoT Platform console, click your target Enterprise Edition instance to view the device status and Message Forwarding logs. For more information, see Cloud-side operational logs.

2. After the data is pushed, navigate to the Data Editor tab of the destination data table in the Tablestore console to check whether the specified data was received. For more information, see Query time series data.

   You can use the SQL query feature to perform complex queries and efficient analysis on data in Tablestore. For more information, see Use SQL to query time series data.

Examples

• Device time series data ingestion solution

• Time series analytics and storage solution

Related documents

• For instructions on how to connect devices to IoT Platform and report data, see Device connection guide.

• On the instance details page, check the available resources for Message Forwarding TPS to ensure you have enough capacity to forward device messages to Tablestore. For more information, see View instance information and operational data. If the available resources are insufficient, you can upgrade the instance. For more information, see Upgrade an instance.

• For information about the usage notes and frequently asked questions (FAQs) about Tablestore, see General questions about Tablestore.