Example of parsing Protobuf data-IoT Platform (Deprecated)(IoT)-阿里云帮助中心

This topic describes how to use a data parsing task to parse and store Protobuf data reported by a device.

Prerequisites

Create a product and a device. Obtain the device certificate, which includes the ProductKey, DeviceName, and DeviceSecret. For more information, see Create a product and Create a single device.
The development environment is installed and configured.
This example uses the Python Link software development kit (SDK) to develop a device that reports Protobuf data. The development environment is a Linux operating system (Ubuntu 20.04 64-bit). For information about how to install Python and configure the SDK, see Environment requirements and configuration.

Note

The steps in this example are performed with regular user permissions. If you perform an operation that requires administrator permissions, run the command with sudo.

Install Protobuf

Log on to your development environment.
Run the following command to install the Protobuf tool.
```
pip install protobuf
```

Run the following commands to download and decompress the Protobuf compiler package.

Note

The compiler package version must match the installed Protobuf version. You can find the corresponding compiler package for the version number returned in the previous step. For more information, see Protobuf Versions.

wget https://github.com/protocolbuffers/protobuf/releases/download/v24.2/protoc-24.2-linux-x86_64.zip
unzip protoc-24.2-linux-x86_64.zip

The following files are extracted:

bin/protoc
include/google/protobuf/any.proto
include/google/protobuf/api.proto
include/google/protobuf/compiler/plugin.proto
include/google/protobuf/descriptor.proto
include/google/protobuf/duration.proto
include/google/protobuf/empty.proto
include/google/protobuf/field_mask.proto
include/google/protobuf/source_context.proto
include/google/protobuf/struct.proto
include/google/protobuf/timestamp.proto
include/google/protobuf/type.proto
include/google/protobuf/wrappers.proto
readme.txt

Run the following command to view the PIP installation path.
```
which pip
```
The command returns the following output in this example:
```
/usr/bin/pip
```
Run the following command to copy the bin/protoc file to the PIP installation path.
```
cp bin/protoc /usr/bin
```

Run the following command. If the command returns output similar to the following, the compiler is installed successfully.

protoc

root@xxx:~# protoc
Usage: protoc [OPTION] PROTO_FILES
Parse PROTO_FILES and generate output based on the options given:
  -IPATH, --proto_path=PATH  Specify the directory in which to search for
                             imports.  May be specified multiple times;
                             directories will be searched in order.  If not
                             given, the current working directory is used.
                             If not found in any of the these directories,
                             the --descriptor_set_in descriptors will be
                             checked for required proto file.
  --version                  Show version info and exit.
  -h, --help                 Show this text and exit.
  --encode=MESSAGE_TYPE      Read a text-format message of the given type
                             from standard input and write it in binary
                             to standard output.  The message type must
                             be defined in PROTO_FILES or their imports.
  --deterministic_output     When using --encode, ensure map fields are
                             deterministically ordered. Note that this order
                             is not canonical, and changes across builds or
                             releases of protoc.

Generate .desc and .data files for the sample data

This example shows how a device reports Protobuf data over a custom topic. A .desc file is used to read binary data from a .data file. The data is then parsed into the JSON format so that a data parsing task can process and store it.

The JSON data parsed in this example is as follows:

Note

Data parsing tasks do not support parsing array data.

{
  "pname": "test",
  "pid": 1234,
  "pemail": "test@example.com"
}

In a Linux operating system, you can use the installed Protobuf compiler to generate the required files.

Write a .proto file to define the structure for parsing the fields in the sample JSON data.

Run the following command to create and open the book.proto file.
```
vim book.proto
```

Enter the following content, then save and exit.

syntax = "proto3";

package tutorial;

message Person {
  optional string pname = 1;
  optional int32 pid = 2;
  optional string pemail = 3;
}

message AddressBook {
  repeated Person people = 1;
}

Run the following command to generate the corresponding book.desc file.
```
protoc  --descriptor_set_out=./book.desc ./book.proto
```
Run the following command to generate a callable Python class file named book_pb2.py.
```
protoc --python_out=.  book.proto
```
Create a person.py file that uses the structure defined in the book.proto file. In this file, use person.SerializeToString() to serialize the data into a binary format and write the output to the person.data file.
1. Run the following command to create and open the person.py file.
```
vim person.py
```
2. Enter the following content, then save and exit.
```
import book_pb2
person = book_pb2.Person()
person.pid = 1234
person.pname = "test"
person.pemail = "test@example.com"
with open('person.data', "wb") as f:
    f.write(person.SerializeToString())  
   
```
3. Run the following command to execute person.py and generate the binary data file person.data.
```
python3 person.py
```
Download and save the generated book.desc and person.data files to your computer.

Configure a data parsing task

In this example, the Protobuf data reported by the device is parsed and then stored in a custom storage table.

In the IoT Platform console, on the Overview page, click the target Enterprise instance card.
Create a custom storage table: Set Display Name and Identifier to testProtoc. For the other settings, you can use the default or custom values.
Create a data parsing task. Set the task name to testProtoc.
Configure the source node.
1. Set Data Source Type to IoT Instance Topic and Topic Category to Custom Topic. Then, click Next.
  In this example, set Product to Street Light, Device to All Devices, and Topic Name to /k-****/$(deviceName)/user/update.
2. Set Topic Data Format to Protobuf. Upload the book.desc and person.data files that you saved. Then, click Verify Parsing.
  After the data is parsed, you can view it in the Parsing Preview section.
  Set Message Type to tutorial.Person. The parsing preview shows fields including pname: test, pid: 1234, and pemail: test@example.com.
3. Click Save.
Configure data filtering. Configure the task to output field data from the source node when the value of pid is 1234.
Configure the destination node. Store the output data in the custom storage table named testProtoc.
Set Destination Type to IoT Instance Custom Storage Table. Configure three VARCHAR fields (precision: 1024 each):
- pname: metric name pname, set as unique key
- pid: metric name pid
- pemail: metric name pemail
Start the data parsing task.

Develop a device to connect and report data

Return to your Linux operating system and develop a device that connects to IoT Platform and reports Protobuf data.

Run the following command to create and open the device program file mqtt_pub.py.
```
cd ~
vim mqtt_pub.py
```

Enter the following code, then save and exit.

Note

In the code, replace the placeholder values for product_key, device_name, and device_secret with the actual values from your device certificate. You also need to replace the ProductKey in the lk.publish_topic path /gs*****/device1/user/update.

import sys
from linkkit import linkkit
import logging
import os.path

# config log
__log_format = '%(asctime)s-%(process)d-%(thread)d - %(name)s:%(module)s:%(funcName)s - %(levelname)s - %(message)s'
logging.basicConfig(format=__log_format)

lk = linkkit.LinkKit(
    host_name="cn-shanghai",
    product_key="gs*****",
    device_name="device1",
    device_secret="06aec**************728f56"
     )
lk.enable_logger(logging.DEBUG)


def on_device_dynamic_register(rc, value, userdata):
    if rc == 0:
        print("dynamic register device success, value:" + value)
    else:
        print("dynamic register device fail, message:" + value)


def on_connect(session_flag, rc, userdata):
    print("on_connect:%d,rc:%d" % (session_flag, rc))
    pass


def on_disconnect(rc, userdata):
    print("on_disconnect:rc:%d,userdata:" % rc)


def on_topic_message(topic, payload, qos, userdata):
    print("on_topic_message:" + topic + " payload:" + str(payload) + " qos:" + str(qos))
    pass


def on_subscribe_topic(mid, granted_qos, userdata):
    print("on_subscribe_topic mid:%d, granted_qos:%s" %
          (mid, str(','.join('%s' % it for it in granted_qos))))
    pass


def on_unsubscribe_topic(mid, userdata):
    print("on_unsubscribe_topic mid:%d" % mid)
    pass


def on_publish_topic(mid, userdata):
    print("on_publish_topic mid:%d" % mid)

def read_into_buffer(filename):
    buf = bytearray(os.path.getsize(filename))
    with open(filename, 'rb') as f:
        f.readinto(buf)
    return buf

                              
lk.on_device_dynamic_register = on_device_dynamic_register
lk.on_connect = on_connect
lk.on_disconnect = on_disconnect
lk.on_topic_message = on_topic_message
lk.on_subscribe_topic = on_subscribe_topic
lk.on_unsubscribe_topic = on_unsubscribe_topic
lk.on_publish_topic = on_publish_topic


lk.config_device_info("Eth|03ACDEFF0032|Eth|03ACDEFF0031")
lk.connect_async()
lk.start_worker_loop()
buf = read_into_buffer('person.data')
print(buf)


while True:
    try:
        msg = input()
    except KeyboardInterrupt:
        sys.exit()
    else:
        if msg == "1":
            lk.disconnect()
        elif msg == "2":
            lk.connect_async()
        elif msg == "3":
            rc, mid = lk.subscribe_topic([(lk.to_full_topic("user/receive/from/jiexi"), 1)])
           
            if rc == 0:
                print("subscribe multiple topics success:%r, mid:%r" % (rc, mid))
            else:
                print("subscribe multiple topics fail:%d" % rc)
        elif msg == "4":
            rc, mid = lk.publish_topic("/gs*****/device1/user/update", buf)
            if rc == 0:
                print("publish topic success:%r, mid:%r" % (rc, mid))
            else:
                print("publish topic fail:%d" % rc)
        else:
            sys.exit()

Run the following commands to activate virtual environments.
```
cd work_dir
source test_env/bin/activate
```
Run the following commands to run the device program.
```
cd ~
python3 mqtt_pub.py
```
The following data is read from the device:
```
bytearray(b'\n\x04test\x10\xd2\t\x1a\x10test@example.com')
```

Enter 4 to trigger the device to report a message.

4
2023-09-01 11:26:46,149-****** - Paho:client:_easy_log - DEBUG - Sending PUBLISH (d0, q1, r0, m1), 'b'/gs*****/device1/user/update'', ... (43 bytes)
publish topic success:0, mid:1
2023-09-01 11:26:46,172-****** - Paho:client:_easy_log - DEBUG - Received PUBACK (Mid: 1)
2023-09-01 11:26:46,172-****** - linkkit:linkkit:debug - DEBUG - post_message :'on_publish' 
2023-09-01 11:26:46,172-****** - linkkit:linkkit:debug - DEBUG - post_message success
2023-09-01 11:26:46,173-****** - linkkit:linkkit:debug - DEBUG - thread runnable pop cmd:'on_publish'
2023-09-01 11:26:46,173-****** - linkkit:linkkit:debug - DEBUG - __on_internal_publish message:1
on_publish_topic mid:1

View data parsing results

Return to your instance in the IoT Platform console. On the IoT Platform logs tab of the Monitoring > Simple Log Service page, you can view logs to confirm that the device is online and reporting data.
In the log records with business type Device-to-Cloud Messages, click View in the Message Content column to view the data parsing results.
On the Offline Storage > Custom Storage Tables tab of the DataService Studio > Data Storage page, find the testProtoc table and click Data Preview in the Actions column to view the stored data.
The preview shows four fields: pemail, pid, pname, and ts, with sample data test@example.com, 1234, test, and 1693555891758.