Kubernetes container log collection

Learn how to collect, process, and analyze Kubernetes container logs with SLS and LoongCollector. This topic covers core concepts, deployment modes, collection workflows, and best practices.

Features

SLS provides the following capabilities for Kubernetes container log collection:

Multi-source log support
- Collects various log types: stdout, stderr, and container text file logs.
Granular container filtering
- Include or exclude containers based on namespace name, pod name, container name, container labels, or environment variables.
Advanced log processing
- Collect multi-line logs: Merge multi-line entries such as Java stack traces into single log events.
- Pre-process logs: Drop invalid data with a Data filtering plugin, or mask and structure sensitive data with Data masking and encryption plugins and Field processing plugins plugins.
- Parse fields: Structure raw logs before storage with Data parsing plugins-based parsing plugins.
Intelligent metadata association
- Automatically associate metadata with container logs, such as container name, image, pod, namespace, and environment variables.
Reliability
- A checkpoint mechanism records the current collection position to ensure log integrity.
- Handles logs when a container stops by providing different strategies for various container runtimes.

Limitations

Container runtime: Only Docker and Containerd are supported.
Docker:
- Requires permissions to access docker.sock.
- For standard output collection, only the json-file logging driver is supported.
- Only the overlay and overlay2 storage drivers are supported. For other storage drivers, you must mount the log directory by using a volume.
Containerd:
- Requires permissions to access containerd.sock.

Multi-line log limit:
The last collected line is buffered for 3 seconds by default to prevent multi-line logs from being split by output delays. Adjust this with the BeginLineTimeoutMs parameter (minimum: 1,000 ms).
Standard output:
The default maximum log entry size is 512 KB (524,288 bytes), with an upper limit of 8 MB (8,388,608 bytes). To increase this limit, set the max_read_buffer_size environment variable on the LoongCollector container.
Important
Do not enable stdout and stderr collection simultaneously. This may cause log entries to interleave incorrectly.

Collection workflow overview

Prepare the log source: Identify the stdout or text file logs to collect.
Install LoongCollector: Deploy the collector to transmit logs to SLS.
Configure collection rules: Define log collection rules and parsing plugins.
Query and analyze logs: Monitor your services with collected log data.

Key processes

Log source and mount point requirements

For standard output logs, LoongCollector automatically discovers the log file path based on container metadata.
For container text file logs, LoongCollector mounts the host root directory to /logtail_host by default. Manual mounting is typically unnecessary. If you use a custom mount point, ensure it meets these requirements:
Custom mount point requirements
Log file path:
- Do not use symbolic links:
  - Incorrect configuration: /var/log -> /mnt/logs.
  - Correct configuration: Use the physical path /mnt/logs directly.
- Mount path matching rule: If your application container's data directory is mounted by using a volume, the collection path must be the same as or a subdirectory of the mount point.
```
1Mount point: /var/log/service
2✅ Valid collection path: /var/log/service or /var/log/service/subdir
3❌ Invalid collection path: /var/log (The path is not specific enough)
```
- Avoid using shared network storage, such as NAS or OSS, as it may cause data truncation, inconsistencies, or collection failures. Use block storage instead.

Install the collector

LoongCollector supports two deployment modes:

Deployment mode: DaemonSet or sidecar.

DaemonSet mode: Deploys a LoongCollector agent on each node automatically. Suitable for most scenarios.
- In DaemonSet mode, the deployment method depends on your cluster type and SLS account relationship.
  - For Container Service for Kubernetes (ACK) clusters, the loongcollector-ds component is pre-integrated. Enable it in the ACK console to complete installation. By default, logs are stored in the SLS project of the account that owns the cluster. Install LoongCollector on Kubernetes.
  - To collect ACK cluster logs into an SLS project under a different Alibaba Cloud account, manually install LoongCollector and configure it with the destination account's ID or AccessKey. Install LoongCollector on Kubernetes.
  - For self-managed clusters, manually install LoongCollector and configure it with the destination Alibaba Cloud account's ID or AccessKey. Install LoongCollector on Kubernetes.
  LoongCollector must be installed before log collection can begin. Collect container logs from a Kubernetes cluster by using a CRD (standard output/file).
Sidecar mode: Injects a LoongCollector container into each application pod. Use this mode for serverless containers, high-volume pods that exceed DaemonSet capacity, or clusters with secure container runtimes. Collect Kubernetes pod text logs (sidecar mode).

Collection rules

SLS supports two methods for defining collection rules:

Configuration method	Features	Use cases	Notes
Kubernetes CRD	Native Kubernetes integration: Declare configurations as CRDs for seamless Kubernetes API integration. Configuration as Code: Supports GitOps workflows and enables version control. Real-time updates: The Operator watches for changes and syncs them to LoongCollector automatically.	Recommended for production clusters and CI/CD environments.	Use only one method to manage a given collection configuration. Mixing methods can invalidate the configuration. If multiple configurations target the same file, enable Allow multiple collections for a file. Otherwise, only one configuration takes effect at random. Alternatively, use data manipulation to process and store multiple copies of a log.
Simple Log Service console	Simple operation: Graphical, no-code configuration. Quick verification: Fast testing and validation. Centralized management: View all configurations in a unified console.	Best for small clusters, debugging, or non-production environments.

Key concepts

Kubernetes: An open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications.
Standard output, standard error, and text file logs: Stdout captures normal program output (business logs, operational records). Stderr captures errors and warnings (stack traces, startup failures). Both are directed to the terminal and captured by the container engine. Text file logs are written to files by applications (for example, Nginx access.log) and are deleted when the container is destroyed unless persisted with a volume.
Checkpoint mechanism: Records the collection position in a file, saved in /tmp/logtail_checkpoint by default. Ensures reliable collection after LoongCollector restarts or node failures.
LoongCollector (Logtail): A high-performance log collector developed by Alibaba Cloud that supports DaemonSet and sidecar deployment in Kubernetes. LoongCollector is the successor to Logtail and is fully backward-compatible.
Kubernetes CRD: A CustomResourceDefinition that lets you define custom resources for configuration. SLS uses AliyunPipelineConfig as its CRD type.
Collection configuration: Defines rules for log types, collection paths, filtering, parsing, and storage location. What is a collection configuration?.
Parsing plugin: A processing unit within the processing plugin configuration for structuring, splitting, filtering, or desensitizing log content. Supports regular expression, separator, JSON, and multi-line modes.

How it works

A user creates a Custom Resource (CR) by using kubectl to define a collection rule.
The loongcollector-operator continuously watches for changes to CRs in the cluster.
When a change is detected, the Operator converts the CR into a LoongCollector configuration and applies it to Simple Log Service.
The LoongCollector agent periodically sends heartbeats to Simple Log Service to fetch configuration updates, pulls the latest collection configuration, and applies it dynamically.
The loongcollector-ds agent collects logs according to the new configuration and sends them to SLS through the configured endpoint.

DaemonSet mode

Deploys a LoongCollector agent on each node to collect logs from all containers on that node. Offers simple operations, low resource consumption, and flexible configuration, but weaker tenant isolation.

How the DaemonSet mode works

In DaemonSet mode, Kubernetes runs exactly one LoongCollector container on each node to collect logs from all containers on that node.
Kubernetes automatically creates and destroys LoongCollector containers as nodes join or leave the cluster, eliminating manual instance management.

Sidecar mode

Injects a LoongCollector sidecar alongside the application container in each pod. The application's log directory is shared via a Kubernetes volume (emptyDir, hostPath, or PVC), allowing LoongCollector to read log files directly. Provides strong tenant isolation and high performance, but consumes more resources.

How the sidecar mode works

In sidecar mode, each pod runs a dedicated LoongCollector container. Log collection is isolated between pods.
A shared volume must be mounted to both the application and LoongCollector containers for log file access.
When a pod's log volume exceeds DaemonSet capacity, sidecar mode lets you allocate dedicated resources to LoongCollector.
Serverless environments lack nodes, making DaemonSet inapplicable. Sidecar mode integrates directly with serverless architectures.

Container discovery

LoongCollector must identify running containers on the node before collecting their logs. This process is called container discovery.

LoongCollector communicates directly with the container runtime daemon on the node, not the cluster's kube-apiserver. This avoids extra load on the kube-apiserver.
LoongCollector accesses the container runtime socket (Docker or Containerd) on the host. It supports including or excluding containers by namespace, pod name, pod labels, or environment variables.

Standard output collection

LoongCollector automatically identifies the correct API or logging driver for each container runtime (Docker, Containerd) based on metadata. It reads the stdout stream directly without accessing container file systems.

LoongCollector periodically saves collection progress to a checkpoint file and resumes from the last position after a restart.

Container text file log collection

Kubernetes isolates container file systems, so a collector cannot directly access files in other containers. LoongCollector mounts the host's root file system to indirectly access application container files.
By default, the host root file system is mounted to /logtail_host. Manual mounting is typically unnecessary. For example, if a container log file is at /log/app.log and its host path is /var/lib/docker/containers/<container-id>/log/app.log, LoongCollector reads it from /logtail_host/var/lib/docker/containers/<container-id>/log/app.log.

Multi-line log parsing

LoongCollector uses a user-defined regular expression to match the beginning of a log line.

Successful match: The line is treated as the start of a new log entry.
Failed match: The line is appended to the current log entry.

When another line matches the start-of-line regular expression, this completes the current log entry and begins a new one.

Log processing when a container stops

Runtime	Destruction latency risk	Log integrity	Optimization
Docker	When a container is stopped, LoongCollector immediately releases the container's file handle, allowing the container to exit normally.	If collection is delayed before the container stops, for example, due to network latency or high resource usage, some logs generated just before the stop may be lost.	Increase the log sending frequency (decrease `flush_interval`).
Containerd	If collection is delayed, for example, due to network latency or high resource usage, the application container may not be destroyed promptly.	When a container is stopped, LoongCollector continues to hold the file handles for the container, keeping the log files open until all content has been sent.	Configure `max_hold_buffer_size` to limit memory usage.

Container metadata retrieval

LoongCollector retrieves Kubernetes metadata directly through the CRI (Container Runtime Interface) API, enabling non-intrusive, real-time metadata tagging during collection.

Docker: LoongCollector uses the Docker Client to communicate with the Docker Daemon for metadata retrieval. The main APIs include:
- ContainerList: Lists running containers on the node.
- ContainerInspect: Returns detailed container configuration and status.
- Events: Listens for container lifecycle events in real time.
Key metadata fields retrieved through the Docker Client:
- LogPath: The host path of the container's stdout log file.
- GraphDriver.Data: The host path of the container's rootfs, used for file system access and troubleshooting.
Containerd: LoongCollector uses the CRI to support containerd and CRI-O environments, collecting metadata from underlying runtimes such as runc or Kata Containers.
- The CRI provides the host path of the stdout log file but not the container rootfs path. LoongCollector uses these methods to find it:
  - File path search: Searches the host file system for the container rootfs path using the container ID.
  - Direct containerd interaction: LoongCollector can bypass the CRI and communicate with containerd directly to obtain the rootfs path and other metadata not available through the CRI.

Best practices

Unified query across environments

To query logs across environments (for example, testing and production), use one of these methods:

Store all environment data in the same Logstore and add tags to distinguish environments. Collect container logs from a cluster by using the console (standard output/file).
Collect data into separate Logstores or projects. Create a StoreView for cross-Logstore queries. This method has no additional storage cost but is read-only and does not support alerting. Use a tag field to identify each log entry's source Logstore.
(Recommended) Collect data into separate Logstores or projects. Use data manipulation to copy selected data to a centralized Logstore. Supports pre-storage parsing and alerting, but is a paid feature.

Collecting logs from multiple sources

Each collection configuration targets one source. Create a separate configuration for each log source.

Granular collection and multi-tenant isolation

In a multi-tenant environment, use separate projects to isolate data. Data across projects is inaccessible from one another, and each project can have independent access permissions.

Automated operations and CI/CD integration

Use the CRD method to incorporate collection configurations into GitOps or IaC workflows for automated, traceable log collection management.