Six common pitfalls that cause log collection failures

Background information

A well-designed log management strategy helps preserve historical data, reduce performance overhead, and improve collection efficiency. However, several common anti-patterns can prevent modern log collectors like LoongCollector from working effectively. By optimizing log management from the source and adopting standard practices, you can achieve efficient and stable log collection.

Using copy truncate causes data loss or duplication

The copy truncate mode for log rotation works by copying the original log file and then truncating it. This approach has several drawbacks:

1. Because the file system inode changes, the collector might misidentify the copied file as new and collect it again.

2. Logs generated between the non-atomic copy and truncate operations can be lost. This content is not in the copied file and is erased by the truncate operation.

3. The truncate operation can shrink the file size or alter its header content. A smaller file or a changed file signature can cause the collector to misidentify the file as new, leading to duplicate collection.

As a result, the copy truncate mode can lead to duplicate log collection, data loss, or inconsistent data.

Recommended solution:

We recommend using the create mode for log rotation. This approach creates a new file and renames the old one, which ensures file integrity and continuity. If you cannot avoid using copy truncate, specify the exact path name in your collection configuration.

Incomplete collection from NAS or OSS storage

Network Attached Storage (NAS) typically uses an eventual consistency model, which is a common design in distributed systems. In real-time collection scenarios, this can cause the following problems:

1. Inconsistent file metadata and content. Due to eventual consistency, metadata such as file size may be updated before the actual content is written.

2. Reading a file hole. When metadata indicates that a file has grown but the content has not yet been synchronized, a read operation may return \0 characters (a file hole).

3. Data latency. The results of a write operation may not be immediately visible to a read operation, causing collection delays.

4. Data loss. Because NAS does not support inotify and has poor list performance, the collector may fail to discover new files, leading to data loss.

As a result, the collected data may not match the final content of the file.

Recommended solution:

We recommend using EBS or a local disk on self-managed machines to ensure the efficiency and consistency of log reads and writes. If this is not possible, design your consumer side to handle potential log anomalies.

Multi-process writing causes incomplete data

Concurrently writing to the same log file from multiple processes is a common but discouraged practice. It can lead to the following problems:

1. Interleaved content. Writes from multiple processes can become interleaved, resulting in garbled log entries.

2. Incomplete collection. When a write event occurs, the collector begins to read the file. If another process writes to the file during this collection, the newly written content may be skipped.

3. File lock contention. Multiple processes writing to the same file can cause file lock contention, which impacts write performance and reliability.

This practice can result in collected data that is incomplete and inconsistent with the final file content.

Recommended solution:

We recommend that each process writes to its own separate file to ensure log integrity and order. If this is unavoidable, design your consumer side to handle potential log anomalies.

Creating file holes causes data loss or duplication

Creating a hole at the beginning of a file to free up space is a risky practice for the following reasons:

1. File signature changes. To prevent data loss from inode reuse, LoongCollector (formerly iLogtail) uses the content at the beginning of a file as part of its unique file signature. Creating a hole can alter this signature, causing the collector to misidentify the file as new.

2. Data integrity issues. Creating a file hole replaces existing content with\0 characters, which can lead to the loss of important historical logs.

3. File system fragmentation. Frequently creating file holes can lead to file system fragmentation, which degrades read and write performance.

Recommended solution:

We recommend using a standard log rotation mechanism, such as the logrotate utility, to manage log file sizes. This ensures log integrity and traceability. If you must create a file hole, use fallocate instead of truncate or dd, and design your consumer side to handle potential log anomalies.

Frequent overwrites lead to inconsistent data

Frequently overwriting an entire log file is an unsafe practice that can cause the following problems:

1. Inconsistent metadata and content. During an overwrite, file metadata like size may be updated before the content, causing the collector to read incomplete or inconsistent data.

2. Risk of data loss. If an overwrite occurs during log collection, the data read by the collector may be corrupted or lost.

3. Loss of historical data. Frequent overwrites prevent the retention of historical logs, which complicates troubleshooting and analysis.

This practice can lead to inconsistent data collection or even complete data loss.

Recommended solution:

We recommend using append writes to record logs and a log rotation mechanism to manage file size. If this is unavoidable, design your consumer side to handle potential log anomalies.

Saving files with vim causes duplicate collection

When you edit and save a file using vim, its save mechanism can cause the following issues:

1. Inode changes. Vim may create a new file and replace the original, resulting in a different inode. This can cause the collector to misidentify the file as new.

2. File signature changes. The header content of the new file may differ from the original, altering the file signature and preventing the collector from correctly identifying it.

3. Data loss. When vim replaces a file, the logging application may not switch its file handle to the new file, causing subsequent logs to be lost.

This editing method can lead to duplicate log collection or data loss.

Recommended solution:

If you only need to view logs, use read-only tools like less or grep. If you must use vim, design your consumer side with logic for deduplication and error handling.

Summary

Logs are the "black box" of your system, and their quality directly impacts troubleshooting efficiency and system reliability. By avoiding the anti-patterns discussed in this document and following best practices like standard log rotation, local disk storage, and single-process append writes, you can significantly reduce log collection risks and improve observability.