Collect IoT and Embedded Device Logs

C Producer runs on the device and sends the collected log data to Simple Log Service over the network. The following figure shows the log data collection solution for IoT devices.

Embedded development requirements

An IoT or embedded engineer needs deep development skills. When you operate a massive fleet of devices, the ability to manage, monitor, and diagnose black box devices matters just as much. Embedded development has the following main log requirements:

• Data collection — How do you collect data in real time from millions or tens of millions of devices spread around the world?

• Debugging — How do you use one solution for both online data collection and real-time debugging during development?

• Online diagnostics — When an online device reports an error, how do you locate the device quickly and view the context that caused the error?

• Monitoring — How many devices are online now? How are their working states and geographical locations distributed? How do you get real-time alerting for faulty devices?

• Real-time data analysis — How do you connect the data that devices generate to real-time computing and big data warehouses to build user personas?

  

Major challenges in the IoT field

IoT adoption is growing fast, and more devices are entering daily life, such as smart routers, TV dongles, Tmall Genie, and robot vacuum cleaners. These devices bring the convenience of smart technology, but the embedded development model of traditional software does not transfer well to them, and traditional device logging solutions cannot meet their requirements.

Almost none of the methods that work for traditional software work in the IoT field. The main challenges come from the following characteristics of IoT devices:

• Large number of devices — In traditional O&M, a company that manages 10,000 servers already counts as a large one. For IoT devices, 100,000 online devices is only a low threshold.

• Wide distribution — Once the hardware is deployed, it is often spread across a whole country or even the whole world.

• Black box — Devices are hard to log on to and debug, and most of the time their state is unknown.

• Limited resources — To control costs, IoT devices ship with limited hardware, such as 32 MB of memory in total. Methods that work on PCs often fail here.

C Producer

C Producer is a lightweight version of Logtail, the Simple Log Service client. C Producer inherits the stability, high performance, and low resource consumption of Logtail, but it does not provide a real-time configuration management mechanism.

An application can create multiple Producers, and each Producer can contain multiple Clients. Each Client has its own destination address, log level, local debugging switch, cache size, custom ID, and topic.

C Producer features

C Producer provides the following features for log collection on IoT and embedded devices:

• Multi-tenancy — You can process several types of logs, such as Metric, DebugLog, and ErrorLog, at different priorities. You can also configure multiple Clients, and each Client has its own collection priority, destination project, and LogStore. C Producer sends the logs of the Client that has the highest priority first.

• Contextual query — Logs from the same Client belong to the same context, so you can view the logs that precede and follow a specific log.

• Concurrent sending and resumable upload — You can set an upper limit for the cache. After the limit is reached, log writes fail.

• High-concurrency client writes — The sending thread pool is configurable and supports writes of hundreds of thousands of logs per second. For measured throughput and resource usage on both server-class and constrained hardware, see Performance testing.

• Direct network sending — Client logs are not written to disk. After the data is generated, it goes directly to the server over the network.

• Separation of computing and I/O — C Producer writes logs asynchronously on the client, so log writes do not block worker threads.

• Low resource consumption — On low-performance hardware such as a Raspberry Pi, generating 100 logs per second has almost no impact on resources.

  C Producer also provides the following features that are exclusive to IoT devices:

• Local debugging — C Producer writes log content to the local device, with settings for rotation, log count, and rotation size. Enable local debugging to test your application on the device when no network connection is available.

• Fine-grained resource control — You can set a separate cache limit and aggregation method for each type of data or log.

• Compressed log caching — C Producer compresses data that fails to be sent before it is cached, which reduces memory usage on the device.

  

C Producer handles the implementation details of log collection. You do not need to implement collection logic in your application, and log collection does not affect the normal operation of your business.

Download and integration

Download the C Producer library from GitHub.

For the detailed installation method and the integration procedure, see the README file in the repository.

C Producer with Simple Log Service

C Producer and Alibaba Cloud Simple Log Service form an end-to-end logging solution for IoT devices. C Producer collects logs on the device and sends them to Simple Log Service, where the logs become available for consumption, query, analysis, and visualization. The solution provides the following characteristics:

• Large scale — Simple Log Service supports real-time writes from hundreds of millions of clients, with a data volume of PB/Day.

• Fast collection — Data can be consumed as soon as it is written, and it becomes visible on the server within seconds.

• Fast queries — A complex query with five conditions processes billions of data records within one second.

• Fast analysis — Complex analysis with five-dimension aggregation and GroupBy aggregates hundreds of millions of data records within one second.

• Broad integration — Simple Log Service integrates with Alibaba Cloud services, and works with storage, computing, and visualization systems in various open source formats.

  Simple Log Service provides the following functions for the collected device logs:

• LogHub — Real-time log collection and consumption.

• Search/Analytics — Query and real-time analysis.

• Interconnection — Grafana and JDBC/SQL92.

• Visualization — Dashboard and report functions.

  

Other collection solutions from Simple Log Service

Logtail, the Simple Log Service client, runs on millions of X86 servers. For technical background, see the following Logtail articles: multi-tenant data isolation and results in the Double 11 shopping festival, order-preserving log collection based on Polling and Inotify. Simple Log Service also provides other collection solutions:

• Mobile SDK — Collects data on the Android and iOS platforms, and already serves tens of millions of DAU per day.

• Web Tracking (JS) — A lightweight collection method that requires no signature.

  For X86 servers, use Logtail. For Android and iOS applications, use the Mobile SDK. For IoT and embedded devices, use C Producer.

Comparison with other embedded collection solutions

The following table compares C Producer with other embedded collection solutions.

Performance testing

C Producer was tested in two environments. Use the X86 results as the reference for server-class hardware and the Raspberry Pi results as the reference for resource-constrained devices.

Environment configuration

• High-performance scenario: a traditional X86 server.

• Low-performance scenario: a Raspberry Pi in a low-power environment.

  The following figure shows the configurations of both environments.

C Producer configuration

• ARM (Raspberry Pi)

  • Cache: 10 MB

  • Aggregation time: 3 seconds (C Producer packages and sends the aggregate data as soon as the aggregation time, the aggregated data packet size, or the aggregated log count is reached)

  • Aggregated data packet size: 1 MB

  • Aggregated log count: 1000

  • Sending threads: 1

  • Custom tags: 5

• X86

  • Cache: 10 MB

  • Aggregation time: 3 seconds (C Producer packages and sends the aggregate data as soon as the aggregation time, the aggregated data packet size, or the aggregated log count is reached)

  • Aggregated data packet size: 3 MB

  • Aggregated log count: 4096

  • Sending threads: 4

  • Custom tags: 5

  Log sample (nine key-value pairs, with a data volume of approximately 350 bytes)

__source__: 192.0.2.1
__tag__:1: 2
__tag__:5: 6
__tag__:a: b
__tag__:c: d
__tag__:tag_key: tag_value
__topic__: topic_test
_file_: /disk1/workspace/tools/aliyun-log-c-sdk/sample/log_producer_sample.c
_function_: log_producer_post_logs
_level_: LOG_PRODUCER_LEVEL_WARN
_line_: 248
_thread_: 40978304

Test results

• Results on the X86 platform

  • C Producer reaches a sending speed of 90 MB/s. It uploads 200,000 logs per second, and uses 70% of the CPU and 140 MB of memory.

  • At 200 logs per second, sending data has almost no impact on the CPU of the server, which drops to less than 0.01%.

  • A client thread takes 1.2 microseconds on average to send one piece of data, that is, to write one log.

  

• Results on the Raspberry Pi platform

  • In the test on the Raspberry Pi, the CPU frequency is only 600 MHz, so the performance is about 1/10 of the performance of the server. The device sends a maximum of 20,000 logs per second.

  • At 20 logs per second, sending data has almost no impact on the CPU of the Raspberry Pi, which drops to less than 0.01%.

  • A client thread takes about 12 microseconds on average to send one piece of data, that is, to write one log. The Raspberry Pi is connected to a PC over USB and shares the network of the PC.

  

For more typical Simple Log Service scenarios, see Yunqi Community.