Logtail

Limits

Limitations

Architecture

• Logtail for Linux supports x86_64 and arm64.

• Logtail for Windows supports x86_32 and x86_64.

Compute resources

• CPU: Reserve at least 0.1 core.

• Memory: A minimum of 30 MB is required for host scenarios, and 150 MB for Kubernetes scenarios. A minimum of 100 MB is required for Kubernetes Controller/Operator.

Actual usage depends on the collection rate, the number of monitored directories and files, and the degree of sending blockage. Ensure that the actual usage is less than 80% of the limit.

System environment

For supported operating systems, see System types.

Kubernetes

• To collect logs using a DaemonSet, Kubernetes 1.10.0 or later is required. The HostToContainer mount propagation must be supported.

• To collect logs using a Custom Resource Definition (CRD) (the alibaba-log-controller component), Kubernetes 1.16.0 or later is required.

  The apiextensions.k8s.io/v1beta1 API provided by Kubernetes 1.7.0 and later also supports CRDs. However, the stability of the API in the beta version depends on the specific Kubernetes version and is not guaranteed.

Docker

The following limits apply to the collection of container standard output:

• You must add "log-driver": "json-file" to the Docker configuration file daemon.json.

• For CentOS 7.4 and later (except CentOS 8.0), you must set fs.may_detach_mounts=1. For more information, see Bug 1468249, Bug 1441737, and issue 34538.

Storage medium

Do not use shared network storage media, such as NAS or OSS. This may cause data truncation, content inconsistency, or collection suspension. Use Elastic Block Storage (EBS).

Limits

Limitations

Size of a single log

The default limit is 512 KB. You can adjust this limit using the max_read_buffer_size startup parameter. The maximum size cannot exceed 8 MB. For more information, see Set Logtail startup parameters.

After a multi-line log is split based on a regular expression that matches the first line, the size of each resulting log is still limited to 512 KB. If a log is larger than 512 KB, it is forcibly split into multiple logs for collection. For example, if a log is 1,025 KB, the first 512 KB is processed, then the next 512 KB, and finally the last 1 KB. This results in multiple incomplete logs.

File encoding

Logtail supports log files that are encoded in UTF-8 or GBK. For optimal processing performance, we recommend that you use UTF-8 encoding.

Log file size

Unlimited.

Log file rotation

The default size of the log rotation queue is 20. You can adjust this using the logreader_max_rotate_queue_size startup parameter. For more information, see Set Logtail startup parameters.

If more than 20 files are not processed, newly generated logs are lost. In this case, first check whether the write quota of the Logstore shard is exceeded, and then adjust the concurrency level of Logtail. For more information, see Recommended parameter values.

For more information, see related technical articles.

Collection behavior when log parsing is blocked

When log parsing is blocked, Logtail keeps the file descriptor for the log file open. This prevents the file from being deleted during the blockage, which would cause data loss.

If the log file is rotated multiple times while parsing is blocked, Logtail places the file in the rotation queue.

Regular expression

Perl-compatible regular expressions are supported.

JSON

Standard JSON, as defined in RFC 7159 and ECMA-404, is fully supported. Non-standard JSON, such as {"name": "\xE5\xAD\xA6"}, is not supported.

Multiple Logtail configurations for one file

By default, a file can match only one Logtail collection configuration. If logs from a file need to be collected multiple times, see How to collect multiple copies of logs from a single file.

File opening behavior

Logtail keeps the collected files and the files that are pending collection in the rotation queue open to ensure data integrity. A file is closed in the following situations:

• The file has not been modified for more than 5 minutes.

• The file is rotated and collection is complete.

• The Logtail collection configuration changes.

If you want the file handle to be released within a controllable time after a file is deleted, regardless of whether collection is complete or logs are still being written, you can set a timeout period using the force_release_deleted_file_fd_timeout startup parameter. For more information, see Set Logtail startup parameters.

Initial log collection behavior

Logtail collects only incremental log files. When a file modification is first detected, if the file size exceeds 1 MB (512 KB for container standard output), collection starts from the last 1 MB of the file. Otherwise, collection starts from the beginning of the file.

You can change the value of the tail_size_kb parameter in a Logtail configuration to change the limit. For more information, see Logtail configurations (Legacy).

If the log file is not modified after the Logtail collection configuration is applied, Logtail does not collect logs from the file. To collect historical files, see Import historical log files.

File overwriting behavior

Logtail identifies a file by its inode and the hash of its first 1,024 bytes. If a file is overwritten and its inode or the hash of its first 1,024 bytes changes, it is treated as a new file and collected from the beginning. Otherwise, it is not collected.

File moving behavior

If a file is moved and it matches a Logtail collection configuration that has never matched this file before, the moved file is treated as a new file and collected from the beginning. Otherwise, it is not collected.

File collection history

Logtail keeps the file collection history in memory to ensure that only incremental parts are collected after a file changes. If logs are written beyond the retention scope, duplicate data collection occurs.

• By default, the history of files from the last month is retained.

• If there are more than 5,000 historical files in the same directory, only records from the last week are retained.

• If there are more than 10,000 historical files in the same directory, only records from the last day are retained.

Non-standard text logs

For log lines that contain \0, versions later than 2.1.10 and 3.0.12 retain only the \0 in the middle of the log and discard the prefix and suffix \0 parts. Other versions might truncate the log at the first \0 or retain it completely. We recommend that you upgrade your Logtail version. For other escape characters, such as ASCII color codes, or non-printable characters, Logtail reports them as is.

Limits

Limitations

Initial log collection behavior

If you collect container standard output, when a file modification is first detected, if the file size exceeds 512 KB, collection starts from the last 512 KB. Otherwise, collection starts from the beginning. You can adjust this using the StartLogMaxOffset parameter in the Logtail collection configuration. For more information, see Collect container standard output in DaemonSet mode using the console.

Symbolic link

When you collect logs from container files, symbolic links for directories and files are not supported.

Container lifecycle

By default, the container lifecycle must be at least 10 seconds for Logtail to collect logs. In container file collection scenarios, for performance reasons, Logtail limits container updates to 10 times within 3 minutes. You can adjust these settings using the docker_config_update_interval and max_docker_config_update_times startup parameters. For more information, see Set Logtail startup parameters.

Standard output log rotation

The container's standard output files are rotated by Docker or kubelet. The default rotation size for kubelet log files is 10 MB (Docker log rotation is set to 100 MB on ACK). If the container's standard output rate is greater than 10 MB/s, files rotate quickly. To avoid data loss, collect logs from container files or modify the kubelet containerLogMaxSize parameter.

Standard output log driver

If you use Docker as a container runtime, you must add "log-driver": "json-file" to the Docker configuration file daemon.json.

Limitations

Limitations

Checkpoint timeout

By default, if a file is not modified for more than 30 days, its checkpoint is automatically deleted. If you set preserve:false in the Logtail configuration, the checkpoint is automatically deleted if the file is not modified for more than 30 minutes.

Checkpoint storage policy

Checkpoints are saved periodically (every 15 minutes) and automatically when the program exits. You can adjust this using the check_point_dump_interval startup parameter. For more information, see Set Logtail startup parameters.

Checkpoint storage location

The default storage path is /tmp/logtail_checkpoint. You can change this path using the check_point_filename startup parameter. For more information, see Set Logtail startup parameters.

Handling during downtime

Because checkpoints are saved periodically, recovery after downtime starts from the last fully saved checkpoint. This may cause duplicate data collection. Adjust the checkpoint storage policy to reduce duplicate data collection.

Limits

Limits

Configuration update latency

It takes about 30 seconds for a configuration update made in the console or using an API to take effect on the Logtail client.

Dynamic loading of configurations

Supported. An update to one configuration does not affect others.

Total number of loadable configurations for a single Logtail instance

Theoretically unlimited. We recommend that you use no more than 100 Logtail collection configurations on a single server.

Third-party flusher output

Configurations created through the Simple Log Service console or an API are associated with a Logstore. Therefore, when you use a third-party flusher in the plugin configuration, Logtail sends a copy of the data to the current Logstore by default.

Multi-account and cross-account

Supported. For more information, see Configure a user identifier and Use Logtail to collect container logs across Alibaba Cloud accounts.

Multi-region

This is not supported by default. If you have this requirement, submit a ticket.

Global Accelerator

Supported. After you enable this feature on the server-side, set up Global Accelerator on the client. For more information, see Enable Global Accelerator.

Limitations

Limitations

Number of machines

Theoretically unlimited. Configure no more than 100,000 machines. Otherwise, heartbeats cannot be obtained correctly.

Number of applied configurations

Theoretically unlimited. We recommend that you apply no more than 1,000 configurations.

Limits

Limitations

Log processing throughput

The default limit for raw log traffic is 20 MB/s. Data is uploaded after being encoded and compressed, with a typical compression ratio of 5 to 10. If this traffic limit is exceeded, data loss may occur. You can adjust this limit using the max_bytes_per_sec startup parameter. For more information, see Set Logtail startup parameters.

Maximum performance

The single-core capabilities are as follows:

• Simple mode: The maximum log processing capacity is 100 MB/s.

• Regex mode: The default maximum processing capacity is 20 MB/s. This depends on the complexity of the regular expression.

• The throughput for separator-delimited logs is capped at 40 MB/s.

• JSON mode: The maximum log processing capacity is 30 MB/s.

Using the process_thread_count startup parameter to set multiple processing threads can increase performance by 1.5 to 3 times.

Maximum number of monitored directories and files

This is related to the mem_usage_limit parameter (default is 384 MB for host scenarios and 2,048 MB for container scenarios) and is divided into the following four levels:

• Maximum total number of monitored directories: ⌊mem_usage_limit/100⌋ × 5,000. This does not include directories specified in the Logtail collection configuration blacklist.

• Maximum total number of monitored directories and files: ⌊mem_usage_limit/100⌋ × 50,000. This does not include directories specified in the Logtail collection configuration blacklist, but includes files not matched by any Logtail collection configuration.

• Total number of monitored directories and files for a single collection configuration: ⌊mem_usage_limit/100⌋ × 5,000. This does not include directories specified in the Logtail collection configuration blacklist, but includes files not matched by the Logtail collection configuration.

• Total number of monitored subdirectories and files in a single directory: ⌊mem_usage_limit/100⌋ × 5,000. This includes directories specified in the Logtail collection configuration blacklist and files not matched by any Logtail collection configuration.

When the number of monitored directories and files at any of these levels reaches the limit, Logtail stops monitoring the remaining directories and files at that level. You can increase the monitoring limit by improving the precision of the monitored directories in the Logtail collection configuration or by adjusting the mem_usage_limit startup parameter. For more information about the mem_usage_limit parameter, see Set Logtail startup parameters.

For Linux systems, Logtail also supports monitoring directories through the inotify mechanism to reduce log collection latency. The maximum number of directories (including subdirectories) that can be monitored by inotify is 3,000.

Resource limit handling policy

If Logtail's resource usage exceeds the maximum limit for 5 minutes, Logtail forcibly restarts. This may cause data loss or duplication.

Multi-tenant data isolation

Logtail provides isolation at the configuration level. An exception in one Logtail collection configuration does not affect others.For more information, see related technical articles.

Log collection latency

Under normal conditions, the latency from when a log is written to disk to when it is collected by Logtail is less than 1 second.

Log upload policy

Logtail automatically aggregates and uploads logs from the same file. Aggregation is triggered when the number of logs exceeds 4,000, the total log size exceeds 512 KB, or the collection time exceeds 3 seconds. The upload is triggered when any of these conditions is met.

Limitations

Limitations

Network error handling

In the case of a network exception, Logtail actively retries and automatically adjusts the retry interval. In extreme cases, there is a risk of duplicate log collection or data loss:

• If a packet sent by the client is received by the server, but the server's response is not received by the client within 15 seconds, the client's retry will cause duplicate data collection.

• If a network link exception causes a packet received by the client to be corrupted, and this error occurs 5 consecutive times, the data is discarded.

Resource quota exceeded

If the data sending rate exceeds the maximum quota of the Logstore, Logtail blocks collection and automatically retries. To resolve this issue, increase the number of shards for the Logstore.For more information, see related technical articles.

Client time exception

If the time difference between the client and the server exceeds 15 minutes, retries may fail up to 5 times, and the data is discarded. To resolve this issue, correct the time on the client machine.

Project or Logstore does not exist

After 5 unsuccessful retries, the data is discarded. This can happen if you deleted the Logstore using an API. To resolve this issue, delete the corresponding Logtail collection configuration and its application using an API.

Authentication failed

After 5 unsuccessful retries, the data is discarded. This issue usually occurs for two reasons:

• The issue occurs when Logtail starts up because of an unstable network, which prevents authentication.

• If the issue persists, the machine cannot establish an HTTPS connection with the Simple Log Service Endpoint to obtain authentication.

Other unknown errors

After 5 unsuccessful retries, the data is discarded.

Maximum retry time before timeout

If data sending fails continuously for more than 6 hours, the data is discarded.

Status self-check

Logtail supports automatic restart in exceptional situations, such as an abnormal program exit or resource usage that exceeds the configured limits.

Exceeding the maximum number of monitored directories and files

Logtail cannot find the collection path promptly and accurately. This may cause data loss.

Significant collection delay

The log collection progress lags behind log generation. If more than 20 unprocessed log files are rotated, data loss occurs.