Troubleshoot LoongCollector collection issues

Updated at:

When collecting data with LoongCollector, you may encounter exceptions. This topic describes how to troubleshoot and resolve them, with examples for common scenarios.

Troubleshoot collection issues

Collection exceptions have complex and varied causes. Different root causes can lead to identical symptoms, and some exceptions are not immediately apparent. This guide explains how to promptly identify, troubleshoot, and diagnose these exceptions by category.

  1. Log on to the Log Service console, and in the Log Application area, click Simple Log Service Data Insight. On the SLS Collection Rule List tab of Ingestion Management, enable the feature as prompted in the dialog box.

    CloudLens for SLS helps you monitor and manage Log Service resources, improve troubleshooting efficiency, and detect and respond to anomalies in multi-dimensional metrics.
  2. Return to the Project List and click the desired Project.

  3. Click the imageLog Storage tab. In the list of Logstores, hover over the desired Logstore, click the Logtail配置管理 icon on the right, and then click Basic Diagnostics.

    Advanced Diagnostics provides a diagnostics dashboard and lets you query exceptions over a longer period. For more information, see Operational Diagnostics and Monitoring.
  4. Refer to Common Collection Error Types to troubleshoot the error shown in the diagnostics. Alternatively, check the client log for details in the ilogtail.LOG and logtail_plugin.LOG files in the /usr/local/ilogtail/ directory.

Common collection errors

Machine group heartbeat failure

See Troubleshooting heartbeat issues.

Logs not collected

If the query page shows no data during log collection, possible causes include:

  1. In the console, check the consumption preview for the Logstore to verify log collection. If logs are collected but queries return no data, it is likely because you have not created an index.

  2. No new logs: If the target log file has no updates, no logs are collected.

  3. Abnormal machine group heartbeat: Troubleshoot heartbeat issues.

  4. No collection configuration: You must create a collection configuration and apply it to a machine group. The system then deploys the configuration to the server. For more information, see Associate a machine group with a collection configuration.

  5. Incorrect collection configuration path: Verify that the log file path in the collection configuration is correct.

    In container scenarios, you can also enable the container metadata preview to view the matched containers.
  6. For log collection from containers, note the following:

    1. Paths with mounts or symbolic links: Symbolic links are not supported. The priority for mount types is EmptyDir > hostPath > NAS (not recommended). For more information, see Requirements for log sources and mount points (Important).

      EmptyDir mount: Provides the best performance. You can log on to the container to view logs. hostPath mount: Allows you to view logs by logging on to the host or the container. Logs are retained even if the container crashes. NAS mount: Not recommended. Collecting logs from NAS has very low performance and is prone to failure. In addition, you can only perform simple grep queries on the collected logs.
    2. Inconsistent labels or environment variables: Verify that the labels and environment variables in the collection configuration match those of the container. To check Docker labels, you must log on to the host and run the docker inspect <container ID> command.

  7. Collection error: If all the above checks pass, troubleshoot collection errors by using diagnostic information in the console.

Repeated data collection

  • High memory usage causes the collector instance to crash repeatedly. After each crash, it re-collects data from the last checkpoint, resulting in data duplication. To resolve this, modify the CPU usage limit or the network sending concurrency limit.

  • An editor, such as vim, may have overwritten the file. Check its creation time to confirm.

    # Method 1: Check the birth time.
    stat your_file 
    # Method 2: If Method 1 does not work, check crtime.
    stat -c %i your_file
    df . # Query the partition.
    sudo debugfs -R 'stat <3014895>' /dev/sda3 # Replace the inode between < and > but keep < and >. Replace sda3 with the device of the partition that you found by running the previous command.
  • Check if multiple collector instances are running.

  • Extremely low log volume (no new logs for more than 15 minutes) can clear the collection progress, leading to duplicate collection.

  • Check for overlapping collection configurations. This is particularly important for stdout, which allows duplicate collection by design.

  • Troubleshoot collection errors by using diagnostic information in the console: The MULTI_CONFIG_MATCH_ALARM error is present.

  • The log collection path constantly changes, and the collection configuration is immediately recreated after being deleted:

    • Check if a symbolic link in the directory is causing duplicate collection.

    • Check if log tags are configured using environment variables or in ACK. If so, remove them.

  • A container restart in a Pod that uses a hostPath volume can cause duplicate collection.

    • The collector finds container logs by detecting containers (similar to docker ps) and then finding their log paths (similar to docker inspect). When a configuration match occurs, the collector reports the collection path and configuration to the main program. When a container restarts, the collector treats it as a new container and reports the path and configuration again. If the Pod uses a hostPath volume for its logs, the previous container's logs persist. The collector then, by default, re-reads the last 1 MB of the log file, which results in duplicate data.

  • Duplicate collection of container logs occurs when the collection path is on NAS.

    • If the NAS directory is slow to register, the log reader might be created after the checkpoint is dumped. This causes the collection progress to be lost. Another symptom is a long delay in adding a new watcher. To resolve it, change the check_point_dump_interval value to 600.

Ingestion latency

There are several types of collection latency:

  • A few seconds of latency is expected. If this is a concern, configure a time plugin.

  • If a large volume of data is delayed, check if the collector has restarted. After a restart, the collector reads data from an earlier point in the log file, which can cause collection latency. Verify that the most recent data is arriving in real time. If so, the system is working correctly.

  • If the latency affects only a single line, check if the last line of the log file is missing a carriage return.

  • Latency on a specific machine can occur if the collector started later than expected. To verify this, check the startup time in the /usr/local/ilogtail/app_info.json file or search the /usr/local/ilogtail/ilogtail.LOG file for "network error" messages.

  • If none of the above scenarios apply, troubleshoot collection errors by using diagnostic information in the console or use CloudLens for SLS to analyze LoongCollector log collection latency.

Collection loss

  • Check if the pod lifecycle is too short (<20s). We recommend writing logs to a hostPath volume for collection.

  • Verify that the env and label configurations match the container's.

  • Check if collection latency is causing incomplete data.

  • If standard output logs that exceed 16 KB are truncated in ACK, ASK, or SAE scenarios, use file-based collection.

  • Collector restarts can also cause incomplete data. Check the update_time in /usr/local/ilogtail/app_info.json to determine if a restart occurred.

    • Active restart: The collector terminates itself when it reaches a configured resource limit. An active restart is characterized by a difference in start times between the collector's daemon process and its worker process. To resolve this, adjust the CPU usage limit or network sending concurrency limit.

    • Passive restart: In a passive restart, k8s kills the collector for exceeding the container's resource limit, or the kernel kills it for exceeding the cgroup limit. A pod restart count greater than one indicates this type of restart. Check for OOM kill events and adjust the CPU usage limit or network sending concurrency limit.

      Note: If you need to adjust resource parameters, this often indicates a large log volume. Ensure that the number of Shards and the resource quota are sufficient.
  • If data is inconsistent across instances, check if different collector versions are the cause. See configuration management to view versions and perform upgrades.

  • Intermittent data loss that recurs after a restart:

    • Use CloudLens for SLS to check if the LogStore has exceeded its limits.

    • Examine the ilogtail.LOG and logtail_plugin.LOG files in the /usr/local/ilogtail/ directory for error messages.

  • If none of the preceding scenarios apply, use diagnostic information in the console to troubleshoot collection errors.

Invalid log format

  • The collection format is incorrect. Review the collection configuration. Use full regex mode or plugin mode to extract fields as described in Continuously collect text logs from a host.

  • A JSON field cannot be parsed or extracted. Use a parsing tool to verify that the log content is a strict JSON object, because the plugin cannot parse non-JSON objects. As a workaround, consider modifying the format of the original field.

  • A missing delimiter can cause an entire log entry to be treated as a single block. To resolve this, learn about index settings and configure them as needed.

FAQ

Unexpected exceptions during log collection

If logs are being collected correctly, your configuration is likely correct. If you have not modified the configuration or added any new collection configurations, consider if you need to tune collection performance and increase your quota.

  • Adjust collection performance: Modify the upper limits for CPU and memory usage.

  • Use CloudLens for SLS for monitoring: In Data Insight, check if the project ingestion traffic exceeds the quota limit. In Collection Monitoring > File Collection Monitoring > Collection File Distribution, check for delayed files. This can happen when the current read rate is lower than the log generation rate. When CPU usage is at its maximum, logs cannot be fully read. As a result, log files remain open, preventing the release of directory space.

Log timestamp issues

  • The __time__ field represents the time when a log is collected. This value can be set by using an SDK when writing logs.

  • The receive_time field in the tag indicates when Log Service receives the log.

  • The time field in a log is just a regular data field.

For example, if you use the sdk to write a log on January 2 but set its date to January 1 (which you can do via the time field or manually), a query will show that receive time is January 2 and __time__ is January 1. Only receive time represents the actual time the log was written. Both __time__ and the time field in the log are user-defined.

Are log timestamps affected by time zones? Log timestamps are independent of time zones because they are stored as Unix timestamps. A Unix timestamp represents the number of seconds or milliseconds that have elapsed since 00:00:00 UTC on January 1, 1970. For example, the Unix timestamp 1620994708 corresponds to Fri May 14 2021 12:18:28 GMT+0000.

Garbled logs

Log Service uses UTF-8 encoding by default. Ensure your raw logs are encoded in UTF-8.

Appendix

Common collection errors

Error code

Description

Solution

LOG_GROUP_WAIT_TOO_LONG_ALARM

Indicates a long delay between when a data packet is generated and when it is sent.

Verify that data is being sent correctly. Check for issues such as the data volume exceeding the default configuration, an insufficient quota, or network problems.

LOGFILE_PERMINSSION_ALARM

The collector does not have permission to read the specified file.

Check the startup account for iLogtail on the server. We recommend running it with root permissions.

SPLIT_LOG_FAIL_ALARM

Failed to split logs into lines because the start-of-line regex did not match the beginning of a log entry.

Verify that the start-of-line regex is correct.

For single-line logs, you can set it to.*.

MULTI_CONFIG_MATCH_ALARM

By default, a file can be matched by only one collection configuration. If multiple configurations match the same file, only one takes effect.

Docker standard output can be collected by multiple configurations.

For a detailed solution, see Use CloudLens to troubleshoot iLogtail duplicate file configuration issues.

REGEX_MATCH_ALARM

In full regex mode, the log content does not match the specified regular expression.

Copy the log sample from the error message and use it to create and test a new regular expression.

PARSE_LOG_FAIL_ALARM

In modes such as JSON or delimiter, parsing fails because the log format does not match the defined structure.

Click the error message to view the detailed error.

CATEGORY_CONFIG_ALARM

The collection configuration is invalid.

A common cause is that the regular expression fails to extract the file path as the Topic.

For a detailed solution, see Use CloudLens to troubleshoot iLogtail collection configuration errors.

LOGTAIL_CRASH_ALARM

The collector crashed because it exceeded the server's resource usage limits.

Modify the CPU and memory usage limits.

REGISTER_INOTIFY_FAIL_ALARM

Failed to register the log listener on a Linux system. This may be caused by insufficient directory permissions or because the directory has been deleted.

Verify that the collector has permission to access the directory and that the directory exists.

DISCARD_DATA_ALARM

Data loss occurred due to insufficient CPU resources or network flow control.

Modify the CPU usage limit or the network sending concurrency limit.

SEND_DATA_FAIL_ALARM

Failed to send data. Possible causes include:

  • An AccessKey has not been created for the Alibaba Cloud account.

  • The client cannot connect to Log Service, or the network link quality is poor.

  • The write quota for the Log Service server is insufficient.

  • Create an AccessKey for your Alibaba Cloud account.

  • Check the local configuration file at/usr/local/ilogtail/ilogtail_config.json and runcurl <server_address> to verify connectivity.

  • Increase the number of shards for the Logstore to support higher data write volumes.

SEND_QUOTA_EXCEED_ALARM

The log write traffic has exceeded the quota.

Increase the number of shards in the console.

READ_LOG_DELAY_ALARM

Log collection is lagging behind log generation. This is typically caused by insufficient CPU resources for the configuration or by network flow control.

Modify the CPU usage limit or the network sending concurrency limit.

When you import historical data, a large amount of data is collected in a short period. You can ignore this error if it occurs in this scenario.

DROP_LOG_ALARM

The system has started dropping logs because collection is lagging behind generation, and more than 20 unprocessed log rotations have occurred. This is typically caused by insufficient CPU resources or network flow control.

Modify the CPU usage limit or the network sending concurrency limit.

LOGDIR_PERMINSSION_ALARM

The collector lacks read permission for the log monitoring directory.

Verify that the log monitoring directory exists. If it exists, check its permission settings.

ENCODING_CONVERT_ALARM

Encoding conversion failed.

Verify that the log encoding format specified in the configuration matches the actual log encoding.

OUTDATED_LOG_ALARM

Outdated log: The log timestamp is more than 12 hours old. Possible causes include:

  • Log parsing is more than 12 hours behind.

  • The user-defined time field is configured incorrectly.

  • The log-generating program is outputting an incorrect time.

Check for a READ_LOG_DELAY_ALARM alarm.

  • If this alarm exists, resolve it by following the solution for READ_LOG_DELAY_ALARM.

  • If this alarm does not exist, check the time field configuration. If the configuration is correct, verify that the log-generating program is outputting the correct time.

STAT_LIMIT_ALARM

The number of files in the directory specified by the collection configuration exceeds the limit.

For a detailed solution, see Use CloudLens to troubleshoot issues with exceeding file or directory limits.

DROP_DATA_ALARM

The process discarded unsaved logs because it timed out upon exiting.

This error is usually caused by a severe collection blockage. Modify the CPU usage limit or the network sending concurrency limit.

INPUT_COLLECT_ALARM

An exception occurred during collection from the input source.

Follow the instructions in the error message.

HTTP_LOAD_ADDRESS_ALARM

The Addresses parameter specified in the HTTP data collection configuration is invalid.

Verify that the Addresses parameter is valid.

HTTP_COLLECT_ALARM

An exception occurred during HTTP data collection.

Troubleshoot based on the error message. This is typically caused by a timeout.

FILTER_INIT_ALARM

An exception occurred during filter initialization.

This is typically caused by an invalid regular expression in the filter. Follow the instructions in the message to fix it.

INPUT_CANAL_ALARM

The MySQL binlog experienced a runtime exception.

Troubleshoot based on the error message. When the configuration is updated, the Canal service may restart. You can ignore errors related to a service restart.

CANAL_INVALID_ALARM

The MySQL binlog has an invalid internal state.

This error is usually caused by inconsistent metadata due to a change in the table's schema at runtime. Confirm whether the table's schema was modified when the error occurred.

MYSQL_INIT_ALARM

An exception occurred during MySQL initialization.

Follow the instructions in the error message.

MYSQL_CHECKPOING_ALARM

Invalid MySQL checkpoint format.

Verify whether the checkpoint-related configuration has been modified.

MYSQL_TIMEOUT_ALARM

MySQL query timed out.

Verify the status of the MySQL server and network.

MYSQL_PARSE_ALARM

Failed to parse MySQL query results.

Verify that the checkpoint format in the MySQL configuration matches the format of the corresponding field.

AGGREGATOR_ADD_ALARM

Failed to add data to the aggregation queue.

Data is being sent too quickly. You can ignore this alarm if the actual data volume is very large.

ANCHOR_FIND_ALARM

An error occurred in the processor_anchor plugin. This can be caused by an incorrect configuration or a log that does not conform to the configuration.

Review the detailed error message and check for problems in the corresponding configuration. The error can be one of the following types:

  • anchor cannot find key: The SourceKey is specified in the configuration, but the corresponding field does not exist in the log.

  • anchor no start: The content corresponding to Start cannot be found in the value of the SourceKey.

  • anchor no stop: The content corresponding to Stop cannot be found in the value of the SourceKey.

ANCHOR_JSON_ALARM

An error occurred in the processor_anchor plugin while expanding the content defined by the Start and Stop parameters as JSON.

Review the detailed error message. Check the content being processed and the related configuration to determine if there is a configuration error or an invalid log.

CANAL_RUNTIME_ALARM

A runtime error occurred in the MySQL binlog plugin.

Review the detailed error message for further investigation. This is generally related to the connected MySQL master.

CHECKPOINT_INVALID_ALARM

Failed to parse the checkpoint.

Review the detailed error message and investigate further based on the checkpoint content (the first 1,024 bytes) and the specific error information.

DIR_EXCEED_LIMIT_ALARM

The number of concurrently monitored directories has exceeded the limit.

Check if the collection configurations for the current Logstore include a large number of directories, and set a reasonable maximum depth for monitoring.

For a detailed solution, see Use CloudLens to troubleshoot issues with exceeding file or directory limits.

DOCKER_FILE_MAPPING_ALARM

Failed to add the Docker file mapping using a command.

Review the detailed error message and investigate further based on the specific error information.

DOCKER_FILE_MATCH_ALARM

The specified file could not be found in the Docker container.

Review the detailed error message and investigate further based on the container information and the file path.

DOCKER_REGEX_COMPILE_ALARM

An error occurred in the service_docker_stdout plugin. Compilation of the BeginLineRegex specified in the configuration failed.

Review the detailed error message and check if the regular expression is correct.

DOCKER_STDOUT_INIT_ALARM

Initialization of the service_docker_stdout plugin failed.

Review the detailed error message. The error can be one of the following types:

  • host...version...error: Check if the Docker Engine specified in the configuration is reachable.

  • load checkpoint error: Failed to load the checkpoint. You can ignore this error if it does not affect ingestion.

  • container...: The specified container has an invalid Label value. Currently, only stdout and stderr are allowed. Check the detailed error for more information.

DOCKER_STDOUT_START_ALARM

The stdout size exceeded the limit during collection by the service_docker_stdout plugin.

This typically occurs when stdout already exists during the first collection. You can ignore this alarm.

DOCKER_STDOUT_STAT_ALARM

The service_docker_stdout plugin could not detect stdout.

This typically occurs when stdout is inaccessible as the container exits. You can ignore this alarm.

FILE_READER_EXCEED_ALARM

The number of concurrently open file objects has exceeded the limit.

This is typically caused by too many files being in the collection state. Check if the collection configuration is reasonable.

GEOIP_ALARM

An error occurred in the processor_geoip plugin.

Review the detailed error message. The error can be one of the following types:

  • invalid ip...: Failed to get the IP address. Check if the SourceKey in the configuration is correct or if there are invalid logs.

  • parse ip...: Failed to parse the city from the IP address. Review the detailed error message for further investigation.

  • cannot find key...: The specified SourceKey could not be found in the log. Check if the configuration is correct or if there are invalid logs.

HTTP_INIT_ALARM

An error occurred in the metric_http plugin. Compilation of the ResponseStringMatch regular expression in the configuration failed.

Review the detailed error message and check if the regular expression is correct.

HTTP_PARSE_ALARM

An error occurred in the metric_http plugin. Failed to get the HTTP response.

Review the detailed error message and check the configuration or the requested HTTP server based on the specific error information.

INIT_CHECKPOINT_ALARM

An error occurred in the binlog plugin. Failed to load the checkpoint. The plugin will ignore the checkpoint and start processing from the beginning.

Review the detailed error message to determine if this error can be ignored.

LOAD_LOCAL_EVENT_ALARM

An error occurred while processing a local event.

This alarm is rare. Review the detailed error message and investigate further based on the file name, configuration name, project, and Logstore.

LOG_REGEX_FIND_ALARM

An error occurred in the processor_split_log_regex and processor_split_log_string plugins. The SplitKey specified in the configuration could not be found in the log.

Review the detailed error message to check for configuration errors.

LUMBER_CONNECTION_ALARM

An error occurred in the service_lumberjack plugin while closing the server when stopping the plugin.

Review the detailed error message for further investigation. This error can typically be ignored.

LUMBER_LISTEN_ALARM

An error occurred in the service_lumberjack plugin during initialization and listening.

Review the detailed error message. The error can be one of the following types:

  • init tls error...: Check the TLS-related configuration based on the specific error message.

  • listen init error...: Check the address-related configuration based on the specific error message.

LZ4_COMPRESS_FAIL_ALARM

An error occurred during LZ4 compression.

Review the detailed error message and investigate further based on values such as log lines, project, category, and region.

MYSQL_CHECKPOINT_ALARM

An error occurred in the MySQL plugin related to the checkpoint.

Review the detailed error message. The error can be one of the following types:

  • init checkpoint error...: Failed to initialize the checkpoint. Check if the checkpoint column specified in the configuration and the retrieved value are correct.

  • not matched checkpoint...: Checkpoint information does not match. Check if the error was caused by manual intervention such as a configuration update. If so, you can ignore the error.

NGINX_STATUS_COLLECT_ALARM

An error occurred in the nginx_status plugin while retrieving the status.

Review the detailed error message and investigate further based on the URL and specific error information.

NGINX_STATUS_INIT_ALARM

An error occurred in the nginx_status plugin. Failed to parse the URL specified in the configuration during initialization.

Review the detailed error message and check if the URL is configured correctly.

OPEN_FILE_LIMIT_ALARM

The number of open files has exceeded the limit, and no new files can be opened.

Review the detailed error message and investigate further based on the log file path, project, and Logstore.

OPEN_LOGFILE_FAIL_ALARM

An error occurred while opening a file.

Review the detailed error message and investigate further based on the log file path, project, and Logstore.

PARSE_DOCKER_LINE_ALARM

An error occurred in the service_docker_stdout plugin. Failed to parse the log.

Review the detailed error message. The error can be one of the following types:

  • parse docker line error: empty line: The log is empty.

  • parse json docker line error...: Failed to parse the log in JSON format. Investigate based on the error message and the first 512 bytes of the log.

  • parse cri docker line error...: Failed to parse the log in CRI format. Investigate based on the error message and the first 512 bytes of the log.

PLUGIN_ALARM

An error occurred during plugin initialization or a related call.

Review the detailed error message. The error can be one of the following types. Investigate further based on the specific error information.

  • init plugin error...: Failed to initialize the plugin.

  • hold on error...: Failed to pause the plugin.

  • resume error...: Failed to resume the plugin.

  • start service error...: Failed to start a service input type plugin.

  • stop service error...: Failed to stop a service input type plugin.

PROCESSOR_INIT_ALARM

An error occurred in the processor_regex plugin. Failed to compile the Regex regular expression specified in the configuration.

Review the detailed error message and check if the regular expression is correct.

PROCESS_TOO_SLOW_ALARM

Log parsing is too slow.

  1. Review the detailed error message to determine if the number of logs, buffer size, and parsing time are normal.

  2. If not, check for other processes consuming excessive CPU resources or for inefficient parsing configurations, such as complex regular expressions.

REDIS_PARSE_ADDRESS_ALARM

An error occurred in the redis plugin. Failed to parse one of the ServerUrls provided in the configuration.

Review the detailed error message and check the URL that caused the error.

REGEX_FIND_ALARM

An error occurred in the processor_regex plugin. The field specified by SourceKey in the configuration could not be found in the log.

Review the detailed error message to check for an incorrect SourceKey configuration or invalid logs.

REGEX_UNMATCHED_ALARM

An error occurred in the processor_regex plugin. A match failed.

Review the detailed error message. The error can be one of the following types. Investigate based on the specific error information.

  • unmatch this log content...: The log content does not match the regular expression in the configuration.

  • match result count less...: The number of matched results is less than the number of Keys specified in the configuration.

SAME_CONFIG_ALARM

Duplicate configurations with the same name exist under the same Logstore. The configuration that is discovered later will be discarded.

Review the detailed error message and check for configuration errors based on the configuration path and other information.

SPLIT_FIND_ALARM

An error occurred in the split_char and split_string plugins. The field specified by SourceKey in the configuration could not be found in the log.

Review the detailed error message to check for an incorrect SourceKey configuration or invalid logs.

SPLIT_LOG_ALARM

An error occurred in the processor_split_char and processor_split_string plugins. The number of parsed fields does not match the number specified in SplitKeys.

Review the detailed error message to check for an incorrect SourceKey configuration or invalid logs.

STAT_FILE_ALARM

An error occurred during file collection by a LogFileReader object.

Review the detailed error message and investigate further based on the file path and error information.

SERVICE_SYSLOG_INIT_ALARM

An error occurred in the service_syslog plugin. Initialization failed.

Review the detailed error message and check if the Address in the configuration is correct.

SERVICE_SYSLOG_STREAM_ALARM

An error occurred in the service_syslog plugin during collection over TCP.

Review the detailed error message. The error can be one of the following types. Investigate based on the specific error information.

  • accept error...: An error occurred during Accept. The plugin will wait for a period and then retry.

  • setKeepAlive error...: Failed to set Keep Alive. The plugin will skip this error and continue.

  • connection i/o timeout...: A timeout occurred while reading over TCP. The plugin will reset the timeout and continue reading.

  • scan error...: A TCP read error occurred. The plugin will wait for a period and then retry.

SERVICE_SYSLOG_PACKET_ALARM

An error occurred in the service_syslog plugin during collection over UDP.

Review the detailed error message. The error can be one of the following types:

  • connection i/o timeout...: A timeout occurred while reading over UDP. The plugin will reset the timeout and continue reading.

  • read from error...: A UDP read error occurred. The plugin will wait for a period and then retry.

PARSE_TIME_FAIL_ALARM

Failed to parse the log time.

  • Check if the time field extracted by the regular expression is correct.

  • Check if the content of the specified time field matches the time expression in the configuration.

For a detailed solution, see Use CloudLens to troubleshoot log time parsing errors.

BINARY_UPDATE_ALARM

A warning about a binary data update.

Refer to the detailed error message for specific information.

CAST_SENSITIVE_WORD_ALARM

An error related to sensitive word type conversion.

Refer to the detailed error message for specific information.

CHECKPOINT_ALARM

A checkpoint-related error.

Refer to the detailed error message for specific information.

CHECKPOINT_V2_ALARM

(For CheckpointManagerV2 only)

Refer to the detailed error message for specific information.

COMPRESS_FAIL_ALARM

Compression failed. The data will be discarded.

Refer to the detailed error message for specific information.

CONFIG_UPDATE_ALARM

An error occurred during configuration fetching or restart.

Refer to the detailed error message for specific information.

DISCARD_SECONDARY_ALARM

(For DiskBufferWriter only)

Refer to the detailed error message for specific information.

DOMAIN_SOCKET_BIND_ALARM

(For ShennongManager only)

Refer to the detailed error message for specific information.

ENCRYPT_DECRYPT_FAIL_ALARM

(For DiskBufferWriter only)

Refer to the detailed error message for specific information.

EPOLL_ERROR_ALARM

Failed to initialize the inotify file descriptor.

Refer to the detailed error message for specific information.

EXACTLY_ONCE_ALARM

Dropped an exactly-once log group because of an invalid sequence ID.

Refer to the detailed error message for specific information.

FUSE_FILE_TRUNCATE_ALARM

A warning about an unconventional FUSE file truncation behavior.

Refer to the detailed error message for specific information.

GLOBAL_CONFIG_ALARM

(For EnterpriseConfigProvider only) Useless configurations were removed.

Refer to the detailed error message for specific information.

HOLD_ON_TOO_SLOW_ALARM

Stopping the module is too slow.

Refer to the detailed error message for specific information.

INNER_PROFILE_ALARM

The file device inode has changed. A new reader will be created.

Refer to the detailed error message for specific information.

INOTIFY_DIR_NUM_LIMIT_ALARM

The inotify watch limit has been exceeded.

Refer to the detailed error message for specific information.

INOTIFY_EVENT_OVERFLOW_ALARM

The inotify event queue has overflowed.

Refer to the detailed error message for specific information.

INVALID_CONTAINER_PATH_ALARM

Failed to set the container base directory because the container log path does not exist.

Refer to the detailed error message for specific information.

INVALID_MEMORY_ACCESS_ALARM

PropagateTimeout accessed an invalid key of mPathWdMap.

Refer to the detailed error message for specific information.

LOG_GROUP_PARSE_FAIL_ALARM

(For DiskBufferWriter only)

Refer to the detailed error message for specific information.

LOG_TRUNCATE_ALARM

The signature is the same but the file size has decreased. Reading starts from the current file size.

Refer to the detailed error message for specific information.

LOGDIR_PERMISSION_ALARM

Failed to open a directory.

Refer to the detailed error message for specific information.

LOGTAIL_CONFIG_ALARM

The ilogtail_config.json file does not exist or failed to be parsed.

Refer to the detailed error message for specific information.

LOGTAIL_CRASH_STACK_ALARM

Provides the stack trace from the last iLogtail crash.

Refer to the detailed error message for specific information.

METRIC_GROUP_PARSE_FAIL_ALARM

(For ShennongManager only)

Refer to the detailed error message for specific information.

MODIFY_FILE_EXCEED_ALARM

The modify cache is at its limit.

Refer to the detailed error message for specific information.

PROCESS_QUEUE_BUSY_ALARM

The log processing queue is full. The modify event is being put back into the event queue.

Refer to the detailed error message for specific information.

READ_STOPPED_CONTAINER_ALARM

The container has stopped.

Refer to the detailed error message for specific information.

REGISTER_HANDLERS_TOO_SLOW_ALARM

Registering handlers took too long.

Refer to the detailed error message for specific information.

RELABEL_METRIC_FAIL_ALARM

Failed to relabel a metric.

Refer to the detailed error message for specific information.

SECONDARY_READ_WRITE_ALARM

(For DiskBufferWriter only)

Refer to the detailed error message for specific information.

SEND_COMPRESS_FAIL_ALARM

(For DiskBufferWriter only)

Refer to the detailed error message for specific information.

SENDING_COSTS_TOO_MUCH_TIME_ALARM

Sending data took too long.

Refer to the detailed error message for specific information.

SERIALIZE_FAIL_ALARM

Serialization failed.

Refer to the detailed error message for specific information.

SKIP_READ_LOG_ALARM

A warning about skipping log reading.

Refer to the detailed error message for specific information.

STREAMLOG_TCP_SOCKET_BIND_ALARM

Indicates a TCP port binding conflict.

Refer to the detailed error message for specific information.

TOO_MANY_CONFIG_ALARM

Too many configurations in the local directory.

Refer to the detailed error message for specific information.

UNEXPECTED_FILE_TYPE_MODE_ALARM

An unexpected file type mode was found.

Refer to the detailed error message for specific information.

USER_CONFIG_ALARM

(For LegacyConfigProvider only) The local configuration is invalid.

Refer to the detailed error message for specific information.

WINDOWS_WORKER_START_HINTS_ALARM

A startup detection alarm for a Windows worker node.

Refer to the detailed error message for specific information.