Common collection errors

LOGFILE_PERMISSION_ALARM

Severity: Warning

Logtail does not have read permission on the target log file.

Cause: The account running Logtail lacks read access to the file.

Solution: Check which account is running Logtail on the server. Restart Logtail as the root user, or grant the Logtail account read permission on the file.

LOGDIR_PERMISSION_ALARM

Severity: Warning

Logtail does not have read permission on the log monitoring directory.

Causes:

• The directory does not exist.

• The directory permissions do not allow access by the Logtail account.

Solution: Confirm that the log monitoring directory exists and that the Logtail account has read permission on it.

REGISTER_INOTIFY_FAIL_ALARM

Severity: Warning

Logtail failed to register an inotify watcher for the log directory on Linux.

Causes:

• Logtail does not have permission to access the directory.

• The directory was deleted.

Solution: Confirm that the directory exists and that Logtail has the required permissions.

OPEN_FILE_LIMIT_ALARM

Severity: Warning

Logtail has reached the open file limit and cannot open additional files.

Solution: Check the error details for the log file path, project, and Logstore. Either reduce the number of files being collected or increase the open file limit on the server.

OPEN_LOGFILE_FAIL_ALARM

Severity: Warning

Logtail failed to open a log file.

Solution: Check the error details for the file path, project, and Logstore. Confirm that the file exists and that Logtail has read permission on it.

FILE_READER_EXCEED_ALARM

Severity: Warning

The number of files Logtail has open at the same time exceeds the limit.

Cause: Too many files are being collected concurrently.

Solution: Review the collection configuration and reduce the number of concurrently collected files.

STAT_FILE_ALARM

Severity: Warning

A LogFileReader object encountered an error while reading data from a file.

Solution: Check the error details for the file path and the specific error message.

SPLIT_LOG_FAIL_ALARM

Severity: Warning

The line-start regular expression does not match the log content. Logtail cannot split the log into separate lines.

Solution: Verify the regular expression. For single-line logs, set the regular expression to .*.

REGEX_MATCH_ALARM

Severity: Warning

In full regular expression mode, the log content does not match the configured regular expression.

Solution: Copy the log sample from the error message and use it to generate a new regular expression.

PARSE_LOG_FAIL_ALARM

Severity: Warning

In JSON or delimiter mode, parsing failed because the log format does not match the defined format.

Solution: Click the error message to view the detailed error report, then correct the format definition to match the actual log format.

ENCODING_CONVERT_ALARM

Severity: Warning

Encoding conversion failed.

Cause: The encoding format in the Logtail configuration does not match the actual encoding of the log file.

Solution: Update the Logtail configuration to use the correct encoding format.

PARSE_TIME_FAIL_ALARM

Severity: Warning

Logtail failed to parse the log timestamp.

Causes:

• The regular expression does not correctly extract the time field.

• The time expression in the Logtail configuration does not match the format of the extracted time field.

Solution:

1. Verify that the regular expression correctly extracts the time field.

2. Verify that the time expression in the configuration matches the actual time format in the log.

OUTDATED_LOG_ALARM

Severity: Warning

Log timestamps are more than 12 hours older than the collection time.

Causes:

• Log parsing is delayed by more than 12 hours.

• The custom time field is misconfigured.

• The log-writing program outputs incorrect timestamps.

Solution:

1. Check whether a READ_LOG_DELAY_ALARM error is also reported. If so, resolve that error first.

2. Verify the time field configuration. If it is correct, check whether the log-writing program outputs valid timestamps.

ANCHOR_FIND_ALARM

Severity: Warning

The processor_anchor plugin encountered an error due to incorrect configuration or non-conforming logs.

Subtypes:

• anchor cannot find key -- The SourceKey parameter is specified, but the corresponding field does not exist in the logs.

• anchor no start -- No match for the Start parameter in the SourceKey field value.

• anchor no stop -- No match for the Stop parameter in the SourceKey field value.

Solution: Check the error details. Verify the SourceKey, Start, and Stop parameters in the configuration.

ANCHOR_JSON_ALARM

Severity: Warning

The processor_anchor plugin failed to expand the JSON content defined by the Start and Stop parameters.

Solution: Check the content being processed and the related configuration for errors or invalid log entries.

REGEX_FIND_ALARM

Severity: Warning

The processor_regex plugin cannot find the field specified by SourceKey in the logs.

Cause: Incorrect SourceKey configuration, or invalid log entries.

Solution: Verify the SourceKey parameter in the configuration.

REGEX_UNMATCHED_ALARM

Severity: Warning

The processor_regex plugin failed to match log content.

Subtypes:

• unmatch this log content... -- The log content does not match the regular expression in the configuration.

• match result count less... -- The number of matched results is fewer than the configured number of Keys.

Solution: Check the error details. Verify that the regular expression and Keys are configured correctly.

SPLIT_FIND_ALARM

Severity: Warning

The split_char or split_string plugin cannot find the field specified by SourceKey in the logs.

Cause: Incorrect SourceKey configuration, or invalid log entries.

Solution: Verify the SourceKey parameter in the configuration.

SPLIT_LOG_ALARM

Severity: Warning

The processor_split_char or processor_split_string plugin parsed a different number of fields than specified in SplitKeys.

Cause: Incorrect SourceKey configuration, or invalid log entries.

Solution: Check the error details and verify the configuration.

LOG_REGEX_FIND_ALARM

Severity: Warning

The processor_split_log_regex or processor_split_log_string plugin cannot find the SplitKey specified in the configuration.

Solution: Check the error details and verify the configuration.

FILTER_INIT_ALARM

Severity: Warning

Filter initialization failed.

Cause: The filter contains an invalid regular expression.

Solution: Correct the regular expression based on the details in the error message.

SEND_DATA_FAIL_ALARM

Severity: Critical

Logtail failed to send data to Simple Log Service.

Causes:

• No AccessKey pair has been created for the Alibaba Cloud account.

• The server running Logtail cannot connect to Simple Log Service, or network quality is poor.

• The write quota on the Simple Log Service server is insufficient.

Solution:

1. Create an AccessKey pair for the Alibaba Cloud account.

2. Check the configuration file at /usr/local/ilogtail/ilogtail_config.json. Run curl <server_address> to verify connectivity.

3. Increase the number of shards for the Logstore to support higher write volume.

SEND_QUOTA_EXCEED_ALARM

Severity: Warning

Log write traffic exceeds the shard quota.

Solution: Increase the number of shards in the console. For more information, see Manage shards.

LOG_GROUP_WAIT_TOO_LONG_ALARM

Severity: Warning

A data packet waited too long in the send queue.

Causes:

• Data volume exceeds the default configuration.

• Insufficient quota.

• Network issues.

Solution: Check whether packets are being sent as expected. Address any data volume, quota, or network issues.

LZ4_COMPRESS_FAIL_ALARM

Severity: Warning

Logtail encountered an error during LZ4 compression.

Solution: Check the error details for the log lines, project, category, and region, then investigate the root cause.

LOGTAIL_CRASH_ALARM

Severity: Critical

Logtail crashed after exceeding the server resource usage limit.

Solution: Increase the CPU and memory usage limits. For more information, see Set Logtail startup parameters.

DISCARD_DATA_ALARM

Severity: Critical

Logtail discarded data due to insufficient CPU resources or network throttling.

Solution: Increase the CPU usage limit or the network sending concurrency. For more information, see Set Logtail startup parameters.

READ_LOG_DELAY_ALARM

Severity: Warning

Log collection is falling behind log generation.

Cause: Insufficient CPU resources for Logtail, or network throttling.

Solution: Increase the CPU usage limit or the network sending concurrency. For more information, see Set Logtail startup parameters.

If you are importing historical data, a large amount of data is collected in a short time. This error is safe to ignore during historical imports.

DROP_LOG_ALARM

Severity: Critical

Log collection is delayed, and the number of unprocessed rotated log files exceeds 20.

Cause: Insufficient CPU resources for Logtail, or network throttling.

Solution: Increase the CPU usage limit or the network sending concurrency. For more information, see Set Logtail startup parameters.

DROP_DATA_ALARM

Severity: Critical

Logtail failed to save logs to the local disk before process exit. Unsaved logs are lost.

Cause: The collection process is severely blocked.

Solution: Increase the CPU usage limit or the network sending concurrency. For more information, see Set Logtail startup parameters.

PROCESS_TOO_SLOW_ALARM

Severity: Warning

Logtail is processing logs too slowly.

Solution:

1. Check the error details for the log quantity, buffer size, and parsing time. Use these values to determine whether the processing speed is acceptable for your data volume.

2. If processing is too slow, check whether other processes on the Logtail node are consuming excessive CPU resources, or whether the parsing configuration contains an inefficient regular expression.

STAT_LIMIT_ALARM

Severity: Warning

The number of files in the collection directory exceeds the limit.

Solution:

1. Check whether the target collection directory contains too many files or subdirectories.

2. Set an appropriate root directory for monitoring and configure a maximum monitoring depth for subdirectories.

3. If needed, increase the mem_usage_limit parameter. For more information, see Set Logtail startup parameters.

DIR_EXCEED_LIMIT_ALARM

Severity: Warning

The number of directories Logtail is monitoring simultaneously exceeds the limit.

Solution: Review the collection configuration for the current Logstore and any other configurations applied to Logtail. Set an appropriate root directory for monitoring and configure a maximum monitoring depth for subdirectories.

AGGREGATOR_ADD_ALARM

Severity: Info

Logtail failed to add data to the queue because data is arriving faster than it can be sent.

Solution: This error is safe to ignore when data volume is large.

CATEGORY_CONFIG_ALARM

Severity: Warning

The Logtail collection configuration is invalid.

Cause: The regular expression failed to extract the file path as a topic.

Solution: Verify the regular expression in the Logtail configuration.

MULTI_CONFIG_MATCH_ALARM

Severity: Warning

Multiple Logtail configurations match the same log file. By default, a log file can match only one configuration. When multiple configurations match, only one takes effect.

Multiple Logtail configurations can be used to collect standard output from Docker containers.

Solution:

• Delete the redundant Logtail configurations.

• Modify the configurations to allow a file to match multiple Logtail configurations. For more information, see How do I collect multiple copies of logs from a file?.

SAME_CONFIG_ALARM

Severity: Warning

A Logtail configuration with the same name already exists for the Logstore. The later-discovered configuration is discarded.

Solution: Check the error details for the configuration path. Remove or rename the duplicate configuration.

PROCESSOR_INIT_ALARM

Severity: Warning

The processor_regex plugin failed to compile the regular expression specified in the Regex parameter.

Solution: Verify that the regular expression is correct.

PLUGIN_ALARM

Severity: Warning

An error occurred during plugin initialization or lifecycle management.

Subtypes:

• init plugin error... -- Plugin initialization failed.

• hold on error... -- Failed to pause the plugin.

• resume error... -- Failed to resume the plugin.

• start service error... -- Failed to start a service input plugin.

• stop service error... -- Failed to stop a service input plugin.

Solution: Check the error details and troubleshoot based on the specific subtype.

DOCKER_FILE_MAPPING_ALARM

Severity: Warning

Logtail failed to add a Docker file mapping.

Solution: Check the error details for the command and error message. Verify the Docker file mapping configuration.

DOCKER_FILE_MATCH_ALARM

Severity: Warning

The specified file cannot be found in the Docker container.

Solution: Check the error details for the container information and file path. Confirm that the file exists inside the container.

DOCKER_REGEX_COMPILE_ALARM

Severity: Warning

The service_docker_stdout plugin failed to compile the BeginLineRegex from the configuration.

Solution: Verify that the regular expression is correct.

DOCKER_STDOUT_INIT_ALARM

Severity: Warning

The service_docker_stdout plugin failed to initialize.

Subtypes:

• host...version...error -- The Docker Engine specified in the configuration is inaccessible. Verify the Docker Engine address.

• load checkpoint error -- The plugin failed to load the checkpoint file. Safe to ignore if it does not affect operations.

• container... -- The label value for the specified container is invalid. Valid values are stdout and stderr.

Solution: Check the error details and troubleshoot based on the specific subtype.

DOCKER_STDOUT_START_ALARM

Severity: Info

The stdout data size exceeds the limit during collection by the service_docker_stdout plugin.

Cause: stdout data already exists when collection starts for the first time.

Solution: This error is safe to ignore.

DOCKER_STDOUT_STAT_ALARM

Severity: Info

The service_docker_stdout plugin cannot detect stdout.

Cause: stdout is inaccessible because the container has exited.

Solution: This error is safe to ignore.

PARSE_DOCKER_LINE_ALARM

Severity: Warning

The service_docker_stdout plugin failed to parse a log line.

Subtypes:

• parse docker line error: empty line -- The log line is empty.

• parse json docker line error... -- Failed to parse the log in JSON format. Check the error message and the first 512 bytes of the log.

• parse cri docker line error... -- Failed to parse the log in CRI format. Check the error message and the first 512 bytes of the log.

Solution: Check the error details and verify the log format configuration.

INPUT_CANAL_ALARM

Severity: Warning

A runtime error occurred in the MySQL binary logging plugin.

The canal service may restart when the configuration is updated. Errors caused by a service restart are safe to ignore.

Solution: Troubleshoot based on the details in the error message.

CANAL_INVALID_ALARM

Severity: Warning

The internal state of the MySQL binary logging plugin is abnormal.

Cause: A table schema change during runtime caused metadata inconsistency.

Solution: Confirm whether the table schema was modified at the time the error occurred.

CANAL_RUNTIME_ALARM

Severity: Warning

A runtime error occurred in the binary logging plugin.

Cause: Typically caused by an issue with the connected primary MySQL instance.

Solution: Check the error details and troubleshoot based on the error message.

MYSQL_INIT_ALARM

Severity: Warning

An error occurred during MySQL initialization.

Solution: Troubleshoot based on the details in the error message.

MYSQL_CHECKPOINT_ALARM

Severity: Warning

The MySQL checkpoint format is abnormal, or a checkpoint error occurred during plugin execution.

Subtypes:

• init checkpoint error... -- The plugin failed to initialize the checkpoint. Check the checkpoint column and retrieved values in the configuration.

• not matched checkpoint... -- The checkpoint information does not match. If this was caused by a configuration update, the error is safe to ignore.

Solution:

1. Confirm whether the checkpoint-related settings in the configuration were modified.

2. Verify that the checkpoint format matches the format of the corresponding field.

MYSQL_TIMEOUT_ALARM

Severity: Warning

A MySQL query timed out.

Solution: Confirm that the MySQL server is reachable and that the network is functioning normally.

MYSQL_PARSE_ALARM

Severity: Warning

Logtail failed to parse MySQL query results.

Cause: The checkpoint format in the MySQL configuration does not match the format of the corresponding field.

Solution: Verify that the checkpoint format matches the field format.

SERVICE_SYSLOG_INIT_ALARM

Severity: Warning

The service_syslog plugin failed to initialize.

Solution: Verify that the Address parameter in the configuration is correct.

SERVICE_SYSLOG_STREAM_ALARM

Severity: Warning

The service_syslog plugin encountered an error during TCP data collection.

Subtypes:

• accept error... -- An error occurred running Accept. The plugin waits and retries.

• setKeepAlive error... -- Failed to set Keep-Alive. The plugin skips this error and continues.

• connection i/o timeout... -- A TCP read operation timed out. The plugin resets the timeout and continues reading.

• scan error... -- A TCP read error occurred. The plugin waits and retries.

Solution: Check the error details and troubleshoot based on the specific subtype.

SERVICE_SYSLOG_PACKET_ALARM

Severity: Warning

The service_syslog plugin encountered an error during UDP data collection.

Subtypes:

• connection i/o timeout... -- A UDP read operation timed out. The plugin resets the timeout and continues reading.

• read from error... -- A UDP read error occurred. The plugin waits and retries.

Solution: Check the error details and troubleshoot based on the specific subtype.

HTTP_LOAD_ADDRESS_ALARM

Severity: Warning

The Addresses parameter in the HTTP data collection configuration is invalid.

Solution: Verify the Addresses parameter.

HTTP_COLLECT_ALARM

Severity: Warning

An error occurred during HTTP data collection.

Cause: Typically caused by a request timeout.

Solution: Troubleshoot based on the details in the error message.

HTTP_INIT_ALARM

Severity: Warning

The metric_http plugin failed to compile the ResponseStringMatch regular expression.

Solution: Verify that the regular expression is correct.

HTTP_PARSE_ALARM

Severity: Warning

The metric_http plugin failed to retrieve the HTTP response.

Solution: Check the configuration and the target HTTP server based on the error message.

INPUT_COLLECT_ALARM

Severity: Warning

An error occurred during data collection from the input source.

Solution: Troubleshoot based on the details in the error message.

GEOIP_ALARM

Severity: Warning

The processor_geoip plugin encountered an error.

Subtypes:

• invalid ip... -- Failed to get an IP address. Verify the SourceKey in the configuration, or check for invalid log entries.

• parse ip... -- Failed to resolve the city from the IP address. Check the detailed error message.

• cannot find key... -- The specified SourceKey does not exist in the logs. Verify the configuration, or check for invalid log entries.

Solution: Check the error details and troubleshoot based on the specific subtype.

NGINX_STATUS_COLLECT_ALARM

Severity: Warning

The nginx_status plugin encountered an error while retrieving status.

Solution: Check the error details for the URL and the specific error message.

NGINX_STATUS_INIT_ALARM

Severity: Warning

The nginx_status plugin failed to parse the URL in the configuration.

Solution: Verify that the URL is configured correctly.

REDIS_PARSE_ADDRESS_ALARM

Severity: Warning

The Redis plugin failed to parse the ServerUrls in the configuration.

Solution: Check the URL that caused the error and correct it.

LUMBER_CONNECTION_ALARM

Severity: Info

The service_lumberjack server was shut down while the plugin was stopping.

Solution: This error is safe to ignore.

LUMBER_LISTEN_ALARM

Severity: Warning

The service_lumberjack plugin encountered an error during listener initialization.

Subtypes:

• init tls error... -- Verify that the TLS configuration is correct.

• listen init error... -- Verify that the address configuration is correct.

Solution: Check the error details and troubleshoot based on the specific subtype.

LOAD_LOCAL_EVENT_ALARM

Severity: Warning

Logtail is handling a local event. This alarm is uncommon.

Solution: If the event was not caused by a manual operation, check the error details for the file name, configuration name, project, and Logstore, then investigate the root cause.

CHECKPOINT_INVALID_ALARM

Severity: Warning

Logtail failed to parse the checkpoint.

Solution: Check the error details for the checkpoint key, checkpoint content (first 1,024 bytes), and the specific error message.

INIT_CHECKPOINT_ALARM

Severity: Warning

The binary logging plugin failed to load the checkpoint file. The plugin ignores the checkpoint and starts processing from the beginning.

Solution: Check the error details and confirm whether reprocessing from the beginning is acceptable.