Configure data storage for custom topics

Prerequisites

• You have a Standard or Premium Enterprise Edition instance. You have purchased the required I/O operations per second (IOPS) and space for time-series storage. For more information, see Purchase an Enterprise Edition instance.

• You have added a custom topic to your device. For more information, see Custom topics.

Use cases

• If you need to store data from a custom topic for querying, you must first create a time-series storage rule. After you create the rule, you can write data from other custom topics to time-series storage by either modifying the storage rule in the console or calling the WriteDevicesHotStorageData API.

• To store data from custom topics for extended periods, reduce storage costs, or perform further data processing such as data analytics, data reports, or data APIs, you can enable Data Backup. This automatically creates a corresponding custom storage table in offline storage. The table name follows the format time-series-storage-rule-${rule ID}-random-ID. If you disable and then re-enable Data Backup, data continues to be written to the previously created custom storage table.

Create a time-series storage rule

1. Log on to the IoT Platform console.

2. On the Instance Overview page, find your Enterprise Edition instance and click the instance name to go to the Instance Details page.

3. In the left-side navigation pane, choose Data Service > Data Storage.

4. On the Time-series Storage tab, click the Storage Rules tab, and then click Create Rule.

5. Complete the following steps in the configuration wizard.

Step 1: Configure basic information

1. Enter a rule name and description.

   Parameter	Description
   Rule name	Enter a name for the rule. The name can contain Chinese characters, letters, Japanese characters, digits, underscores (_), and hyphens (-). It must be 1 to 30 characters long. Each Chinese or Japanese character counts as two characters.
   Rule description	Enter an optional description for the rule.

2. Click Next. If you click Cancel, an empty rule is still created. You can find the rule in the storage rule list and configure its source topics and data processing script later. For more information, see View and update a time-series storage rule.

Step 2: Add a source topic

1. On the Add Source Topic page of the wizard, click Add Topic.

2. In the dialog box that appears, select a product, device, and custom topic as prompted. The topic supports the following wildcards:

   • All devices (+): Specifies all devices under the selected product.

   • user/#: Specifies all custom topics for the selected device.

3. Click OK.

   After you add a custom topic, it appears in the source topic list.

   Important

   You cannot modify a source topic after it has been added. To make changes, Delete the topic and add a new one.

   The Source topics list includes the Message type, Topic, ProductKey, DeviceName, and Actions columns.

4. (Optional) Repeat the previous steps to add multiple source topics.

5. Click Next. If you click Cancel, you can view the list of added source topics on the Rule Details page. For more information, see View and update a time-series storage rule.

Step 3: Edit the data processing script

1. On the Edit Data Processing Script page of the wizard, enter a script in the Edit Script editor. This script parses data from the custom topic and stores the results in time-series storage.

   Important

   Within the same storage rule, the data types for a field's key and value must remain consistent. If a data type changes, for example, from int32 to text, writing new data will fail, and you will no longer be able to query historical data.

2. Click Save at the bottom of the page to finish editing the script.

3. Click Debug at the bottom of the page. This also automatically saves the script draft.

4. In the Debug panel on the right, select a product and device, enter the topic and payload data, and then click Debug to test the script.

   Ensure the topic and payload data you enter are compatible with the script's parsing logic. For information about how to write scripts and for examples, see Script syntax. In this example, the product is set to Thermometer, the topic is user/update, and the payload is in JSON format, where items contains Temperature (value: 38) and Humidity (value: 25).

   The Run Result displays the values of variables declared in the script and the data storage function that was executed. After a successful debug, review the output on the Run Result tab. The following is a sample result:

   action:
     transmit to writeHotStorage[destinationId=0], data :{"25":25,"38":38,"10":10}
   variables:
     c : 10
     data : {"deviceType":"CustomCategory","requestId":"1xxx47","productKey":"gxxxLh","gmtCreate":1626948134445,"deviceName":"E81Hxxxco","items":{"Temperature":{"time":1626948134319,"value":38},"Humidity":{"time":1626948134319,"value":25},"2Co":{"time":1626948134319,"value":10}}}
     t : 38
     h : 25
     fields : {"25":25,"38":38,"10":10}

5. Click Close in the Debug panel.

6. (Optional) Click Publish at the bottom of the page. You can still edit, re-debug, or publish the script.

7. Click Finish and Publish. This closes the configuration wizard and completes the rule setup.

8. Return to the Time-series Storage > Storage Rules tab and click Start for the rule you created. After the rule starts, IoT Platform stores the reported data from the custom topic according to the parsing script.

View and update a time-series storage rule

1. On the Time-series Storage tab, navigate to the Storage Rules tab, find the rule, and click View in the Actions column.

2. On the Rule Details page, view the rule information, including the rule ID, creation time, the number and list of source topics, and data processing script details.

Edit basic rule information

1. Click Edit in the upper-right corner.

2. In the Edit Rule dialog box, modify the Rule name and Rule description.

Update source topics

On the Source topics tab:

• Click Add Topic and follow the instructions in Step 2: Add a source topic to add more source topics.

• In the topic list, click Delete in the Actions column for the target topic to remove it from the list.

Update the data processing script

1. Click the Data processing script tab.

2. Above the script editor, click Edit Draft.

3. Modify the script in the script editor.

4. Click Save, Debug, and then Publish at the bottom of the page.

Optional operations

On the Time-series Storage > Storage Rules tab:

• Stop a rule: Click Stop in the Actions column. After you stop the rule, data from the custom topic will no longer be stored in time-series storage.

• Delete a rule: Click Delete in the Actions column. You must stop a rule before you can delete it.

• Back up data: Turn on the switch in the Data Backup column to write data from the custom topic in time-series storage to a custom storage table. This data can then be used for features like data analytics, data reports, and data APIs.

  Do not delete the custom storage table before you disable Data Backup. You can delete the table only after this feature is disabled.