Integrate with InfluxDB

Updated at:

This topic describes how to use an edge Function Compute application to store device data as time series data in InfluxDB, a popular open source time series database.

Prerequisites

  • Ensure that you have created an edge instance. For more information, see Professional Edition Environment Setup.
  • You have created a light sensor product, a LightSensor device for the product, and assigned the device to an edge instance. For more information, see Sample driver.
  • You have created an edge application that uses a container image. For more information, see Container image application.

    Configure the parameters as follows.

    Table 1. Application information parameters
    ParameterDescription
    Application typeSelect Container Image.
    Repository typeSelect Public Repository.
    Registry AddressSet this to influxdb:latest.
    Application versionSet this to the version number of the database that you use, such as 1.7.9.
    Table 2. Container configuration parameters
    ParameterDescription
    Use host modeSelect No.
    Network port mappingSet two network port mappings as follows:
    • First network port mapping:
      • Host port: 8083
      • Container port: 8083
      • Type: TCP
    • Second network port mapping:
      • Host port: 8086
      • Container port: 8086
      • Type: TCP
    Enable privileged modeSelect Yes.
    Volume mappingSet the volume mapping as follows:
    • Source path: /tmp/influxdb
    • Destination path: /var/lib/influxdb
    • Permissions: Read and write

Background information

In Internet of Things (IoT) scenarios, such as smart buildings, industrial applications, and agriculture, on-site sensors collect real-time data such as temperature, humidity, and air quality. This data is updated over time and is known as time series data. Before you can analyze this data or display it on dashboards, you must store it in a time series database.

IoT Edge provides a Function Compute application feature that helps you quickly store modeled device data in a time series database. No custom development is required. You only need to configure message routing for the Function Compute application and deploy it.

This topic uses a LightSensor device as an example to demonstrate how to store its data in InfluxDB in chronological order.

Step 1: Create an application function

  1. Download the function code package fcApp_influxdb_v1.0.zip.
  2. Log on to the Function Compute console.
    If you have not activated this service, read the terms and select I have read and agree to the terms. Then, click Activate Now.
  3. (Optional) In the navigation pane on the left, click Services & Functions. On the Services & Functions page, in the Service List section, click Create Service to create a service.
    The Service Name parameter is required. Set this parameter to EdgeFC. You can configure the other parameters as needed or use their default settings.
    Note
    • If this is your first time creating a service in Function Compute, follow the configuration wizard.
    • If you have already created the EdgeFC service for other scenarios or miniprogram examples, you do not need to create it again.
  4. After the service is created, on the Services & Functions page, find the EdgeFC service and click Create Function.
  5. On the Create Function page, click Configure and Deploy in the Event Function section.
  6. Set the basic configuration parameters for the function.
    ParameterDescription
    Function Type
    Keep the default option.
    Service
    Select the EdgeFC service that you created.
    Function Name
    Set this to influxdbStore.
    Runtime
    Set the runtime environment for the function and select a method to upload the code. In this example, select Python 3.

    To the right of Upload Code, select Upload Code Package. Click Upload Code and upload the fcApp_influxdb_v1.0.zip package that you downloaded in Step 1.

    Handler
    Use the default value: index.handler.

    You can configure the other parameters as needed or use their default settings. For more information, see Function Compute.

    After you confirm the function information, click Create.

  7. After the function is created, you are redirected to the function details page. On the Code Execution tab, in the Code Execution Management section, select the Online Edit radio button to view the source code.
    在线编辑
    Note The sample code for `influxdbStore` performs the following three steps.
    1. Parses the `ProductKey`, `DeviceName`, and `attribute` from the received device data, which is the `event` parameter.
      event_json = json.loads(event)
      if(event_json["topic"].startswith('/sys'))
        pk, dn, attribute = parseTopic(event_json['topic'])
    2. Parses the payload, which contains the device properties or events.
      
      payload_json = json.loads(event_json["payload"])
    3. Encapsulates the property or event data into the InfluxDB storage format. For more information about the data format, see Appendix: Data format definition in this topic. This step uses the InfluxDB client API for Python.

Step 2: Assign the function to an edge instance

  1. Log on to the IoT Edge console.
  2. In the navigation pane on the left, click Application Management.
  3. Create a Function Compute edge application that uses the function that you created in Step 1. For more information, see Function Compute application.

    The application information parameters are as follows:

    ParameterDescription
    Application nameSet a name for your application, such as influxdbStore.
    Application typeSelect Function Compute.
    RegionSelect the region where you created the service.
    ServiceSelect the EdgeFC service.
    FunctionSelect the influxdbStore function.
    AuthorizationSelect AliyunIOTAccessingFCRole.
    Application versionSet a version for the application. The version number must be unique for this application. You cannot set the same version number for two versions of the same application.

    The following describes the function configuration:

    ParameterDescription
    Enable default configurationsSelect No.
    Run modeThere are two run modes. Select Keep running. The program runs immediately after it is deployed.
    Timeout (seconds)The maximum processing time for a function after it receives an event. Use the default value of 5 seconds. If the function does not return a result within this time, the Function Compute program is forcibly restarted.
    Scheduled runUse the default configuration: Off.
    Environment variableClick Add Environment Variable and add the environment variables shown in the Environment variable configuration table below.
    Table 3. Environment variable configuration
    NameValue
    hostThe endpoint of InfluxDB.
    portThe access port of InfluxDB. The default value is 8086.
    usernameThe username for InfluxDB. The default value is root.
    passwordThe password for InfluxDB. The default value is root.
    databaseThe database name in InfluxDB. The default value is example.

    You do not need to configure any other parameters.

  4. In the navigation pane on the left, click Edge Instances.
  5. Find the edge instance that you created in the "Prerequisites" section and click View.
  6. On the Instance Details page, go to the Edge Applications tab and click Assign Application.
  7. In the Assign Application panel, find the influxdbStore application that you created, click Assign in the Actions column, and then click Close.

Step 3: Configure message routing

This topic explains how to add messages and describes each parameter. For more information, see Set message routing.

  1. On the Instance Details page, select the Message Routing tab and click Add Route to add a message route from the LightSensor device to Function Compute.
  2. Set the following parameters. Then, click Confirm.
    ParameterDescription
    Route NameSet a name for the message route.
    Message SourceSelect Device. Then, select Light Sensor > LightSensor.
    Topic FilterSelect All.
    Message TargetSelect Edge Application and the EdgeFC/influxdbStore function.

Step 4: Deploy the edge instance

  1. On the Instance Details page, click Deploy in the upper-right corner. In the dialog box that appears, click OK to deploy resources, such as sub-devices and functions, to the edge.
  2. Log on to your gateway using Secure Shell (SSH). Run the tail -f /linkedge/run/logger/fc-base/influxdbStore/log.INFO command to view the function logs and monitor its status.
    代码片段

Appendix: Data query

After the data is stored in InfluxDB, you can query it using the InfluxDB API. The following examples show how to query data. For more information, see the InfluxDB API documentation.

  • Query by product to retrieve the values of a property at all points in time. The response lists all stored property values in chronological order.
    curl -G 'http://localhost:8086/query?db=example&u=root&p=root' --data-urlencode "epoch=ms" --data-urlencode 'q=SELECT * FROM "Unique property identifier (unique within the product)" WHERE "_productKey" = $pk and "_dataType" = $type' --data-urlencode 'params={"pk":"ProductKey of the product", "type": "property"}'
    
    // Response
    {
        "results": [{
            "statement_id": 0,
            "series": [{
                "name": "temp",
                "columns": ["time", "_dataType", "_deviceName", "_productKey", "value"],
                "values": [
                    [1570782663561, "property", "0987654321", "ProductKey of the product", 100.1],
                    [1570782688527, "property", "0987654321", "ProductKey of the product", 100.2],
                    [1570782851708, "property", "0987654321", "ProductKey of the product", 104.4]
                ]
            }]
        }]
    }
  • Query by device to retrieve the values of a property at all points in time. The response lists all stored property values in chronological order.
    curl -G 'http://localhost:8086/query?db=example&u=root&p=root' --data-urlencode "epoch=ms" --data-urlencode 'q=SELECT * FROM "Unique property identifier (unique within the product)" WHERE "_deviceName" = $dn and "_dataType" = $type' --data-urlencode 'params={"dn":"Device name", "type": "property"}'
  • Query by device to retrieve the values of a property within a specific time range. The response lists all stored property values within that time range in chronological order.
    curl -G 'http://localhost:8086/query?db=example&u=root&p=root' --data-urlencode "epoch=ms" --data-urlencode 'q=SELECT * FROM "Unique property identifier (unique within the product)" WHERE "_productKey" = $pk and "_dataType" = $type and "time" > $timestamp' --data-urlencode 'params={"pk":"ProductKey of the product", "type": "property", "timestamp": "2019-10-11T09:03:15.611Z"}'

Appendix: Data format definition

The following shows the storage format in InfluxDB for device property data and event-type data.

  • Single-value property
    [
        {
          "measurement": "temp",            // Property name
          "tags": {                         // Property tags
            "_productKey": "1234567890",    // ProductKey from the device certificate
            "_deviceName": "0987654321",    // DeviceName from the device certificate
            "_dataType"  : "property"       // Type: property indicates a property, and event indicates an event
          },
          "time": 1346846400000,            // Timestamp in milliseconds
          "fields": {
            "value": 123                    // Property value
          }
        }
        ...
    ]
  • Struct property
    [
        {
          "measurement": "temp",            // Property name
          "tags": {                         // Property tags
            "_productKey": "1234567890",    // ProductKey from the device certificate
            "_deviceName": "0987654321",    // DeviceName from the device certificate
            "_dataType"  : "property"       // Type: property indicates a property, and event indicates an event
          },
          "time": 1346846400000,            // Timestamp in milliseconds
          "fields": {
            "aa": 12,                       // Property value
            "bb": 12,                       // Property value
            "cc": 12,                       // Property value
            "dd": 12,                       // Property value
          }
        }
        ...
    ]
  • Event
    [
        {
          "measurement": "event123",        // Event name
          "tags": {                         // Event tags
            "_productKey": "1234567890",    // ProductKey from the device certificate
            "_deviceName": "0987654321",    // DeviceName from the device certificate
            "_dataType"  : "event"          // Type: property indicates a property, and event indicates an event
          },
          "time": 1346846400000,            // Timestamp in milliseconds
          "fields": {
            "speed" : 20.8,
            "level" : 4,
            "direction" : "East",
          }
        }
        ...
    ]