Collect IoT and embedded logs

Embedded development requirements

For IoT and embedded engineers, managing, monitoring, and diagnosing black-box devices is as crucial as having deep development expertise. Key requirements for embedded development include the following:

• Data collection: Collect data in real time from millions or tens of millions of devices distributed globally.

• Debugging: Use a single solution for both online data collection and real-time development debugging.

• Online diagnostics: Quickly locate a faulty online device and view the contextual information that led to the error.

• Monitoring: Track the number of online devices, their operational status, and geographical distribution. Provide real-time alerting for faulty devices.

• Real-time data analysis: Integrate device data with real-time computing and data warehouses to build user personas.

Major challenges in the IoT field

When considering solutions to these problems, traditional software development methods are often ineffective in the IoT field. The main challenges arise from the unique characteristics of IoT devices:

• Large number of devices: In traditional Operations and Maintenance (O&M), managing 10,000 servers is considered a large scale. For IoT, 100,000 online devices is just a starting point.

• Wide distribution: Once deployed, devices are often spread across a country or even globally.

• Black box: Devices are difficult to log on to and debug. They are often in an unknown state.

• Resource constraints: For cost reasons, IoT devices have limited hardware, such as only 32 MB of total memory. This makes traditional PC methods ineffective.

C Producer

This is a customized solution for log data collection in Simple Log Service.

The Simple Log Service client, Logtail, is deployed on millions of X86 servers. For more information, see Multi-tenant data isolation and Double 11 practices and Log collection with guaranteed order using a Polling and Inotify combination. Simple Log Service also provides other data collection solutions:

• Mobile SDK: Collects data from Android and iOS platforms, with tens of millions of daily active users (DAUs).

• Web Tracking (JS): A lightweight, signature-free collection method similar to Baidu Analytics.

The Simple Log Service team developed C Producer, a log data collection solution tailored to the features of IoT devices.

C Producer features

The C Producer library inherits the stability, high performance, and low resource consumption of Logtail. It can be considered a lightweight version of Logtail. Although it lacks the real-time configuration management of Logtail, it provides approximately 70% of its other features, including the following:

• Multi-tenancy support: Prioritize and process different types of logs, such as Metric, DebugLog, and ErrorLog. You can configure multiple clients, each with its own collection priority, destination Project, and Logstore.

• Contextual query support: Logs generated by the same client are in the same context. This lets you view related logs that occurred before and after a specific log entry.

• Concurrent sending and resumable upload: Set a cache limit. If the limit is exceeded, log writing will fail.

In addition, C Producer also offers the following features specifically for IoT devices:

• Local debugging: Output logs to a local file. Supports log rotation, log count limits, and rotation size settings.

• Fine-grained resource control: Set different cache limits and aggregation methods for different types of data or logs.

• Compressed log caching: Compress and cache data that failed to send. This reduces the device's memory usage.

Advantages

As a solution tailored for IoT devices, C Producer has clear advantages in the following areas:

• High-concurrency client writes: A configurable sending thread pool supports writing hundreds of thousands of logs per second. For more information, see the Performance Testing section.

• Low resource consumption: Writing 200,000 logs per second consumes only 70% of the CPU. On low-performance hardware, such as a Raspberry Pi, generating 100 logs per second has almost no impact on resources.

• Client-side logs are not written to disk: Data is sent directly to the server-side over the network after it is generated.

• Separation of client-side computing and I/O logic: Logs are output asynchronously and do not block worker threads.

• Multiple priority levels: Configure different priorities for different clients to ensure high-priority logs are sent first.

• Local debugging: Set up local debugging to test your application locally when the network is unavailable.

The C Producer library simplifies your application development steps. You do not need to worry about the implementation details of log collection or its impact on your business operations. This significantly lowers the barrier to data collection.

The following table compares the C Producer solution with other embedded collection solutions.

C producer + Simple Log Service solution

You can use C Producer with Alibaba Cloud Simple Log Service to create a complete logging solution for IoT devices.

• Large scale:

  • Supports real-time writes from hundreds of millions of clients.

  • Supports petabytes of data per day.

• High speed:

  • Fast collection: Zero-latency writes. Data can be consumed immediately after being written.

  • Fast queries: Process billions of data records in less than a second with a complex query (five conditions).

  • Fast analysis: Aggregate hundreds of millions of data records in less than a second with a complex analysis (five-dimension aggregation + GroupBy).

• Broad integration:

  • Seamlessly integrates with various Alibaba Cloud products.

  • Fully compatible with various open source storage, computing, and visualization systems.

Download and use

Download URL: Github

An application can create multiple Producers. Each Producer can contain multiple Clients. For each Client, you can separately configure the destination address, log level, local debugging settings, cache size, custom ID, and topic.

For detailed installation and operation steps, see the README file.

Performance Testing

The configurations are as follows:

C Producer configuration:

• ARM (Raspberry Pi):

  • Cache: 10 MB

  • Aggregation time: 3 seconds (Data is packaged and sent when the aggregation time, aggregated packet size, or aggregated log count is reached)

  • Aggregated packet size: 1 MB

  • Aggregated log count: 1000

  • Sending threads: 1

  • Custom tags: 5

• X86:

  • Cache: 10 MB

  • Aggregation time: 3 seconds (Data is packaged and sent when the aggregation time, aggregated packet size, or aggregated log count is reached)

  • Aggregated packet size: 3 MB

  • Aggregated log count: 4096

  • Sending threads: 4

  • Custom tags: 5

__source__: 192.0.2.1
__tag__:1: 2
__tag__:5: 6
__tag__:a: b
__tag__:c: d
__tag__:tag_key: tag_value
__topic__: topic_test
_file_: /disk1/workspace/tools/aliyun-log-c-sdk/sample/log_producer_sample.c
_function_: log_producer_post_logs
_level_: LOG_PRODUCER_LEVEL_WARN
_line_: 248
_thread_: 40978304
LogHub: Real-time log collection and consumption
Search/Analytics: Query and real-time analysis
Interconnection: Grafana and JDBC/SQL92
Visualized: dashboard and report functions

For more information about typical Simple Log Service scenarios, see the Yunqi Community.