In a containerized environment, application logs are often scattered across the standard output (stdout) or log files of individual Docker containers, making centralized management and quick searches challenging. LoongCollector from SLS aggregates these logs into a single Logstore, enabling centralized storage, structured parsing, data masking, filtering, and efficient query and analysis.
Requirements
Permission requirements: The Alibaba Cloud account or RAM user you use for deployment must have the
AliyunLogFullAccesspermission.Docker and LoongCollector requirements:
If your Docker Engine version is v29.0 or later, or the minimum supported Docker API version is 1.42 or later, you must use LoongCollector 3.2.4 or later. Otherwise, LoongCollector cannot collect standard container output or file logs.
LoongCollector versions 3.2.4 and later support Docker API versions from 1.24 to 1.48.
LoongCollector versions 3.2.3 and earlier support Docker API versions from 1.18 to 1.41.
Limitations on collecting standard container output:
You must add
"log-driver": "json-file"to the Docker configuration file daemon.json.For CentOS 7.4 and later versions, excluding CentOS 8.0, you must set
fs.may_detach_mounts=1.
Limitations on collecting text logs: Only the overlay and overlay2 storage drivers are supported. If you use other driver types, you must manually mount the log directory.
Collection configuration
Prerequisites: Create a project and a logstore. A project is a resource management unit that isolates logs from different applications, and a logstore stores logs.
Configure a machine group (install LoongCollector): Install LoongCollector on the servers from which you want to collect logs and add them to a machine group. Use the machine group to centrally manage collection nodes, distribute configurations, and monitor their status.
Create and configure a log collection rule
Global and input configuration: Define the name of the collection configuration, the log source, and the collection scope.
Log processing and structuring: Configure processing rules based on the log format.
Multi-line logs: This applies to single log entries that span multiple lines, such as Java exception stacks or Python tracebacks. Use a regular expression to identify the start of a line, which marks the beginning of each entry.
Structured parsing: Configure parsing plugins, such as regular expressions, separators, or NGINX mode, to extract structured key-value pairs from raw strings. This simplifies subsequent queries and analysis.
Data Filtering: Configure a collection blacklist and content filtering rules to filter out irrelevant log content. This reduces redundant data transmission and storage.
Log categorization: Configure topics and log tags to distinguish logs from different applications, containers, or source paths.
Query and Analysis Configurations: A full-text index is enabled by default and supports keyword searches. We recommend that you enable a field index on structured fields for precise queries and analysis to improve search efficiency.
Validation and troubleshooting: After completing the configuration, verify that logs are collected as expected. If you encounter issues such as no data being collected, heartbeat failures, or parsing errors, see FAQ.
Preparations
Before you collect logs, create a Project and a LogStore. If you already have these resources, you can skip this section and go to Step 1: Configure a machine group (install LoongCollector).
Project creation
LogStore creation
In the Project list, click the name of the target Project.
In the left-side navigation pane, under
, click +.On the Create LogStore page, configure the following core parameters:
Logstore Name: Set a name that is unique within the Project. This name cannot be changed after the LogStore is created.
Logstore Type: Select Standard or Query based on your requirements.
Billing Mode:
Pay-by-feature (Cannot Be Changed): This mode bills you separately for individual resources, including storage, indexing, and read/write operations. This mode is ideal for small-scale scenarios or when your feature usage is unpredictable.
Pay-by-ingested-data: This mode bills you only for the volume of ingested raw data. This mode provides 30 days of free storage and free features such as data processing and data shipping. It is suitable for business scenarios with a data retention period close to 30 days or complex data processing pipelines.
Data Retention Period: Specify the number of days to retain logs. Valid values: 1 to 3,650. A value of 3,650 indicates permanent storage. The default value is 30.
Leave the other settings at their default values and click OK. For more information, see Manage a LogStore.
Step 1: Configure machine group and install LoongCollector
Deploy LoongCollector as a container on the Docker host and add it to a machine group. Use the machine group to centrally manage multiple collection nodes, distribute configurations, and monitor their status.
Pull the image
On a host with Docker installed, run the following command to pull the LoongCollector image. Replace
${region_id}with the region ID of the host's region or a nearby region (such ascn-hangzhou) to improve download speed and stability.# LoongCollector image address docker pull aliyun-observability-release-registry.${region_id}.cr.aliyuncs.com/loongcollector/loongcollector:v3.0.12.0-25723a1-aliyun # Logtail image address docker pull registry.${region_id}.aliyuncs.com/log-service/logtail:v2.1.11.0-aliyunStart the LoongCollector container
Run the following command to start the container. Make sure to correctly mount the required directories and set the required environment variables:
docker run -d \ -v /:/logtail_host:ro \ -v /var/run/docker.sock:/var/run/docker.sock \ --env ALIYUN_LOGTAIL_CONFIG=/etc/ilogtail/conf/${sls_upload_channel}/ilogtail_config.json \ --env ALIYUN_LOGTAIL_USER_ID=${aliyun_account_id} \ --env ALIYUN_LOGTAIL_USER_DEFINED_ID=${user_defined_id} \ aliyun-observability-release-registry.${region_id}.cr.aliyuncs.com/loongcollector/loongcollector:v3.0.12.0-25723a1-aliyunParameters:
${sls_upload_channel}: The log upload channel. Set its value based on your Project's region and the desired network transfer type. For example:Transfer type
Value format
Example
Use cases
internal network transfer
regionIdcn-hangzhouThe ECS instance and the Project are in the same region.
internet transfer
regionId-internetcn-hangzhou-internetThe ECS instance and the Project are in different regions.
The server is on another cloud platform or in a self-built data center.
transfer acceleration
regionId-accelerationcn-hangzhou-accelerationCross-region communication within and outside Chinese mainland.
${aliyun_account_id}: Your Alibaba Cloud account ID.${user_defined_id}: The custom identifier for the machine group, used to bind the collector to the group (for example,user-defined-docker-1). This identifier must be unique within the region.ImportantRequirements:
Correctly configure the three key environment variables:
ALIYUN_LOGTAIL_CONFIG,ALIYUN_LOGTAIL_USER_ID, andALIYUN_LOGTAIL_USER_DEFINED_ID.Mount
/var/run/docker.sockto monitor container lifecycle events.Mount the root directory (
/) to/logtail_hostto allow access to the host file system.
Verify the container status
docker ps | grep loongcollectorExpected output:
6ad510001753 aliyun-observability-release-registry.cn-beijing.cr.aliyuncs.com/loongcollector/loongcollector:v3.0.12.0-25723a1-aliyun "/usr/local/ilogtail…" About a minute ago Up About a minute recursing_shirleyConfigure the machine group
In the left-side navigation pane, choose . Click , configure the following parameters, and then click OK:
Name: Enter a custom name for the machine group, such as
docker-host-group.Machine Group Identifier: Select Custom Identifier.
Custom Identifier: Enter the
${user_defined_id}that you set when starting the container. The value must be an exact match. Otherwise, the association fails.
Verify heartbeat status
Click the name of the new machine group. On the details page, check the :
OK: LoongCollector has successfully connected to SLS.
FAIL: See Troubleshoot heartbeat issues.
Step 2: Create and configure a log collection rule
Define which logs LoongCollector collects, how it parses and filters them, and then apply the configuration to your machine groups.
On the
Logstores page, click the 
icon next to the name of the target logstore to expand it.Click
next to Import Data. In the Quick Data Import dialog box, select a template based on your log source and click Integrate Now:Docker standard output: Select Docker Stdout and Stderr - New Version.
container standard outputcollection supports both new and old templates. We recommend that you use the new template. To learn about the differences between the new and old versions, see Appendix: Comparison of new and old container standard output versions. To use the old template, see Collect standard output from Docker containers (Old version).Docker file logs: Select Docker File - Container.
Configure the Machine Group Configurations, and then click Next:
Scenario: Select Docker Containers.
Move the machine group that you created in Step 1 from the Source Machine Group list to the Applied Machine Group list.
On the Logtail Configuration page, specify the following settings and click Next.
1. Global and input configuration
Before you start, make sure that you have selected a data collection template and bound a machine group. This step defines the configuration name, log source, and collection scope.
Docker standard output
Global Configurations
Configuration Name: Enter a custom name for the collection configuration. The name must be unique within the Project and cannot be changed after creation. The name must meet the following conventions:
It can contain only lowercase letters, digits, hyphens (-), and underscores (_).
It must start and end with a lowercase letter or a digit.
input configuration
Enable the Stdout and Stderr and/or Standard Error switches. By default, both are enabled.
ImportantTo prevent log inconsistencies, avoid enabling standard output and standard error simultaneously.
Docker file logs
Global Configurations:
Configuration Name: Enter a custom name for the collection configuration. The name must be unique within the Project and cannot be changed after creation. The name must meet the following conventions:
It can contain only lowercase letters, digits, hyphens (-), and underscores (_).
It must start and end with a lowercase letter or a digit.
Input Configurations:
File Path Type:
Path in Container: Collect log files from within the container.
Host Path: Collect logs from local services on the host.
File Path: The absolute path of the log files.
Linux: The path must start with a forward slash (
/). For example,/data/mylogs/**/*.logmatches all files with the .log extension in the/data/mylogsdirectory and its subdirectories.Windows: The path must start with a drive letter. For example,
C:\Program Files\Intel\**\*.Log.
Maximum Directory Monitoring Depth: The maximum directory depth that the
**wildcard in the File Path can traverse. The default value is 0, which indicates the current directory only. Valid values: 0 to 1000.We recommend that you set this parameter to 0 and configure the path to the directory that contains the files.
2. Log processing and structuring
Configure processing rules to convert raw, unstructured logs into structured data, which improves query and analysis efficiency. We recommend that you add a log sample before you configure the rules:
On the Logtail Configuration page, in the Processor Configurations section, click Add Sample Log and enter your log content. The system identifies the log format based on the sample and helps generate regular expressions and parsing rules, which simplifies the configuration process.
Scenario 1: Process multi-line logs
By default, multi-line logs, such as Java exception stack traces, are split into separate entries, resulting in a loss of context. To prevent this, enable multi-line mode and configure a first-line regular expression to merge consecutive lines into a single, complete log entry.
Example:
Raw log | Default mode: Stack trace is broken into multiple entries | Multi-line mode: Stack trace is merged into a single entry |
|
|
|
Configuration steps: In the Processor Configurations section of the Logtail Configuration page, enable Multi-line Mode:
Type: Select Custom or Multi-line JSON.
Custom: If the raw log format is not fixed, you must configure a Regex to Match First Line to identify the starting line of each log entry.
Regex to Match First Line: You can generate this automatically or enter it manually. The regular expression must match a complete line. For the example above, the expression is
\[\d+-\d+-\w+:\d+:\d+,\d+]\s\[\w+]\s.*.Automatic generation: Click Generate Regular Expression, select the required log content in the Log Sample text box, and then click Generate Regex.
Manual input: Click Manually Enter Regular Expression. After you enter the expression, click Validate.
Multi-line JSON: If all raw logs are in standard JSON format, Log Service automatically handles line breaks within a single JSON log.
Processing Method If Splitting Fails:
Discard: If a text segment does not match the first-line rule, it is discarded.
Retain Single Line: Unmatched text is split and retained in the original single-line mode.
Scenario 2: Structure logs
Log Service provides various parsing plugins that convert raw logs into structured data. This provides a solid foundation for subsequent analysis, monitoring, and alerting.
Example:
Raw log | Structured log |
| |
Configuration steps: In the Processor Configurations section of the Logtail Configuration page:
Add a parsing plugin: Click Add Processor and configure a plugin such as regular expression parsing, delimiter parsing, or JSON parsing based on your log format. For example, to collect NGINX logs, select .
NGINX Log Configuration: Copy the complete
log_formatdefinition from your NGINX server's configuration file (nginx.conf) and paste it into this text box.Example:
log_format main '$remote_addr - $remote_user [$time_local] "$request" ''$request_time $request_length ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent"';ImportantThe format definition must be exactly the same as the one used to generate the logs.
Common Parameters: The following parameters appear in multiple data parsing plugins and have consistent functions and usage.
Source Field: The name of the source field to parse. The default value is
content, which is the entire content of the collected log.Keep Source Field on Parse Failure: We recommend that you enable this option. If parsing fails due to a format mismatch, the original log content is retained in the specified source field.
Keep Source Field on Parse Success: Select this option to retain the original log content even if the log is successfully parsed.
3. Log filtering
Collecting low-value or irrelevant logs, such as those at the DEBUG or INFO level, wastes storage, increases costs, impairs query performance, and creates data leakage risks. You can implement fine-grained filtering policies for efficient and secure log collection.
Content filtering
Filter logs based on field content, such as collecting only logs where the level is WARNING or ERROR.
Example:
Raw log | Collect only |
| |
Configuration steps: In the Processor Configurations section of the Logtail Configuration page:
Click Add Processor and select :
Field Name: The log field to use for filtering.
Field Value: The regular expression to use for filtering. Only full-text matching is supported. Partial keyword matching is not supported.
Collection blacklist
Use a blacklist to exclude specified directories or files, preventing the upload of irrelevant or sensitive logs.
Configuration steps: On the Logtail Configuration page, in the section, enable Collection Blacklist and click Add.
Supports full path matching and wildcards (*and?) for directory and file names.
File Path Blacklist: The file paths to ignore. For example:
/home/admin/private*.log: Ignores all files in the/home/admin/directory that start with private and end with .log./home/admin/private*/*_inner.log: Ignores files that end with _inner.log in directories that start with private under the/home/admin/directory.
File Blacklist: The filenames to ignore during collection. For example:
app_inner.log: Ignores all files namedapp_inner.log.
Directory Blacklist: The directory path cannot end with a forward slash (
/). For example:/home/admin/dir1/: The directory blacklist does not take effect./home/admin/dir*: Ignores files in all subdirectories of the/home/admin/directory that start with dir./home/admin/*/dir: Ignores all files in subdirectories named dir at the second level of the/home/admin/directory. For example, files in the/home/admin/a/dirdirectory are ignored, while files in the/home/admin/a/b/dirdirectory are collected.
Container filtering
Set collection conditions based on container metadata, such as environment variables, Pod labels, namespaces, or container names, to precisely control which container logs are collected.
Configuration steps: In the Input Configurations section of the Logtail Configuration page, enable Container Filtering and click Add.
Multiple conditions are combined with a logical AND. All regular expression matching is based on the Go RE2 engine, which has some limitations compared to engines such as PCRE. When you write regular expressions, follow the limits described in Appendix: Regular expression limits (container filtering).
Environment Variable Whitelist/Blacklist: Specify the environment variable conditions for the containers from which you want to collect logs.
K8s Pod Label Whitelist/Blacklist: Specify the label conditions for the Pods where the target containers are located.
K8s Pod Name Regex Match: Specify the containers to be collected by Pod name.
K8s Namespace Regex Match: Specify the containers to be collected by namespace name.
K8s Container Name Regex Match: Specify the containers to be collected by container name.
Container Label Whitelist/Blacklist: Collect logs from containers whose labels meet the specified conditions. This parameter is intended for Docker scenarios and is not recommended for Kubernetes scenarios.
4. Log categorization
In scenarios where multiple applications or instances share the same log format, distinguishing the log source is difficult, which reduces context during queries and impairs analysis efficiency. To address this, you can configure topics and log tags to achieve automated context association and logical categorization.
Topics
If multiple applications or instances have the same log format but different paths, such as /apps/app-A/run.log and /apps/app-B/run.log, it is difficult to distinguish the source of the collected logs. You can generate a topic based on the machine group, a custom name, or file path extraction to distinguish logs from different applications or path sources.
Configuration steps: In the section, select a method to generate topics. The following three methods are supported:
machine group topic: If a collection configuration is applied to multiple machine groups, LoongCollector automatically uses the name of the machine group to which the server belongs as the value of the
__topic__field. Use this method to organize logs by host cluster.Custom: The format is
customized://<custom_topic_name>. For example,customized://app-login. Use this method for static topics with fixed application identifiers.File Path Extraction: Extract key information from the full path of the log file to dynamically tag the log source. Use this method when multiple users or applications share the same log filename but have different paths.
If multiple users or services write logs to different top-level directories but the sub-paths and filenames are the same, the source cannot be distinguished by the filename alone. For example:
/data/logs ├── userA │ └── serviceA │ └── service.log ├── userB │ └── serviceA │ └── service.log └── userC └── serviceA └── service.logYou can configure File Path Extraction and use a regular expression to extract key information from the full path. The matched result is uploaded to the logstore as the topic.
Extraction rules
The number and naming of capturing groups in the regular expression determine the output field format.
In the regular expression for the file path, you must escape the forward slash (
/).Capturing group type
Use case
Generated field
Regex example
Example path
Generated field
Single capturing group (one
(.*?))Distinguishing sources by a single dimension (such as username or environment).
Generates the
__topic__field.\/logs\/(.*?)\/app\.log/logs/userA/app.log__topic__: userAMultiple unnamed capturing groups (multiple
(.*?))Requires multiple dimensions but no semantic labels.
Generates the tag field
__tag__:__topic_{i}__, where{i}is the index of the capturing group.\/logs\/(.*?)\/(.*?)\/app\.log/logs/userA/svcA/app.log__tag__:__topic_1__userA;__tag__:__topic_2__svcAMultiple Capturing Groups - Naming (using
(?P<name>.*？)Requires multiple dimensions and clear field names for easier querying and analysis.
Generates a tag field
__tag__:{name}.\/logs\/(?P<user>.*?)\/(?P<service>.*?)\/app\.log/logs/userA/svcA/app.log__tag__:user:userA;__tag__:service:svcA
Log tagging
Enable log tag enrichment to extract key information from container environment variables or Kubernetes Pod labels and attach it as tags for fine-grained log grouping.
Configuration steps: On the Logtail Configuration page, in the Input Configurations section, enable Log Tag Enrichment and click Add.
Environment Variables: Configure an environment variable name and a tag key. The value of the environment variable is stored as the value of the tag.
Environment Variable Name: The name of the environment variable to extract.
Tag Key: The key for the new tag.
Pod Labels: Configure a Pod label name and a tag key. The value of the Pod label is stored as the value of the tag.
Pod Label Name: The name of the Kubernetes Pod label to extract.
Tag Key: The key for the new tag.
5. Output configuration
By default, all logs are sent to the current logstore, and the lz4 compression method is used. To distribute logs from the same source to different logstores, follow these steps:
Multi-target distribution
LoongCollector 3.0.0 and later supports sending logs to multiple targets. Logtail does not support this feature.
You can configure a maximum of five output targets.
After you configure multiple output targets, the configuration no longer appears in the list for the current logstore. To view, modify, or delete a multi-target distribution configuration, see How do I manage multi-target distribution configurations?.
Configuration steps: In the Output Configurations section of the Logtail Configuration page:
Click
to expand the output configuration.Click Add Output Targets and complete the following settings:
Logstores: Select the destination logstore.
Compression Method: Supported methods are lz4 and zstd.
Route Settings: Route logs based on their tag fields. Logs that match the routing configuration are uploaded to the destination logstore. If the routing configuration is empty, all collected logs are uploaded to the destination logstore.
Tag Name: The name of the tag field for routing. Enter the field name directly, such as
__path__, without the__tag__:prefix. Tag fields are divided into the following two types:For more information about tags, see Manage LoongCollector collection tags.
Agent-related: Tags related to the collection agent itself that do not depend on plugins. Examples include
__hostname__and__user_defined_id__.Input plugin-related: Tags that are provided and enriched by the input plugin. Examples include
__path__for file collection, and_pod_name_and_container_name_for Kubernetes collection.
Tag Value: Logs with a tag value that matches this value are sent to this destination logstore.
Discard this tag?: If enabled, the uploaded logs will not include this tag.
to expand the output configuration.Step 3: Query and analysis configuration
After configuring log processing and plug-ins, click Next to proceed to the Query and Analysis Configurations page:
By default, a full-text index is enabled to support keyword searches on raw log content.
To run precise, field-based queries, click Automatic Index Generation after the Preview Data loads. SLS then creates a field index from the first previewed log entry.
Complete the configuration and click Next to finish the collection setup.
Step 4: Validate and troubleshoot
After you apply the collection configuration to a machine group, the system automatically distributes the configuration and starts collecting incremental logs.
View uploaded logs
Confirm that the log file has new content: LoongCollector collects only incremental logs. Run the
tail -f /path/to/your/log/filecommand and trigger an application operation to ensure new logs are written.Query logs: Go to the Search & Analyze page of the target logstore and click Search & Analyze. The default time range is the last 15 minutes. Check for incoming logs. By default, collected Docker container text logs include the following fields:
Parameter
Description
__source__
The IP address of the LoongCollector (Logtail) container.
_container_ip_
The IP address of the application container.
__tag__:__hostname__
The name of the Docker host where LoongCollector (Logtail) is running.
__tag__:__path__
The log collection path.
__tag__:__receive_time__
The time when the server received the log.
__tag__:__user_defined_id__
The custom ID of the machine group.
Troubleshooting
Machine group heartbeat is FAIL
Check the user ID: If your server is not an ECS instance, or if the ECS instance and the project belong to different Alibaba Cloud accounts, verify that the correct user ID file exists in the specified directory.
Linux: Run the
cd /etc/ilogtail/users/ && touch <uid>command to create a user ID file.Windows: Go to the
C:\LogtailData\users\directory and create an empty file named<uid>.
If a file exists in the specified path and its name matches the Alibaba Cloud account ID of the current project, the user ID is configured correctly.
Check the machine group identifier: If you use a custom ID for your machine group, check whether a
user_defined_idfile exists in the specified directory. If the file exists, verify that its content is consistent with the custom ID configured for the machine group.System
Specified directory
Solution
Linux
/etc/ilogtail/user_defined_id# Configure the custom ID. If the directory does not exist, create it manually. echo "user-defined-1" > /etc/ilogtail/user_defined_idWindows
C:\LogtailData\user_defined_idIn the
C:\LogtailDatadirectory, create a file nameduser_defined_idand write the custom ID to it. If the directory does not exist, create it manually.If both the user ID and the machine group identifier are configured correctly, see Troubleshoot LoongCollector (Logtail) machine group issues for further troubleshooting.
No data is collected
Check for incremental logs: LoongCollector starts collecting data from a file only after new log entries are written to it.
Check the machine group heartbeat status: Go to the page and click the name of the target machine group. Then, in the section, check the Heartbeat status.
If the status is OK, the machine group is connected to the SLS project.
If the status is FAIL, see The machine group heartbeat is FAIL.
Verify that the LoongCollector (Logtail) collection configuration is applied to the machine group.
Go to the page, click the name of the target machine group to go to the Machine Group Configurations page.
Click Manage Configuration. This page displays the All Logtail Configurations list on the left and the Applied Logtail Configs list on the right. If the target LoongCollector (Logtail) configuration is in the right-side list, it is successfully applied to the machine group.
If the target configuration is not in the applied list, click Modify. In the All Logtail Configurations list, select the target configuration, click the
icon to move it to the Applied Logtail Configurations list, and then click OK.
icon to move it to the Applied Logtail Configurations list, and then click OK.Log collection errors or incorrect format
This issue indicates that the network connection and basic configuration are correct. The problem is likely a mismatch between the log content and the parsing rules. Check the specific error message to diagnose the issue:
On the Logtail Configuration page, click the name of the LoongCollector (Logtail) configuration that is reporting errors. On the Log Collection Error tab, click Select Time Range to set the query time range.
In the area, check the error type of the log and see Common error types in data collection for a solution.
Next steps
Log query and analysis: Automatically generate query and analysis statements with the built-in AI-powered generation of query and analysis statements (Copilot).
Data visualization: Monitor key metric trends with visualization dashboards.
Automatic alerting for data anomalies: Set up alert policies to detect data anomalies.
Common commands
View LoongCollector (Logtail) status
docker exec ${logtail_container_id} /etc/init.d/ilogtaild statusView LoongCollector (Logtail) information
docker exec ${logtail_container_id} cat /usr/local/ilogtail/app_info.jsonView LoongCollector (Logtail) running logs
The running logs for LoongCollector (Logtail) are stored in the /usr/local/ilogtail/ directory within the container. The primary log files are loongcollector.LOG and ilogtail.LOG. Rotated log files are compressed into files such as ilogtail.LOG.x.gz. For example:
# View LoongCollector running logs
docker exec a287de895e40 tail -n 5 /usr/local/ilogtail/loongcollector.LOG
# View Logtail running logs
docker exec a287de895e40 tail -n 5 /usr/local/ilogtail/ilogtail.LOGThe output is similar to the following:
[2025-08-25 09:17:44.610496] [info] [22] /build/loongcollector/file_server/polling/PollingModify.cpp:75 polling modify resume:succeeded
[2025-08-25 09:17:44.610497] [info] [22] /build/loongcollector/file_server/polling/PollingDirFile.cpp:100 polling discovery resume:starts
[2025-08-25 09:17:44.610498] [info] [22] /build/loongcollector/file_server/polling/PollingDirFile.cpp:103 polling discovery resume:succeeded
[2025-08-25 09:17:44.610499] [info] [22] /build/loongcollector/file_server/FileServer.cpp:117 file server resume:succeeded
[2025-08-25 09:17:44.610500] [info] [22] /build/loongcollector/file_server/EventDispatcher.cpp:1019 checkpoint dump:succeededRestart LoongCollector (Logtail)
# Stop LoongCollector
docker exec a287de895e40 /etc/init.d/ilogtaild stop
# Start LoongCollector
docker exec a287de895e40 /etc/init.d/ilogtaild startFAQ
Common error messages
Symptom | Cause | Resolution |
| The Project's region does not match the region of the LoongCollector (Logtail) container. | Check the region configuration in |
| The file path is misconfigured. | Verify that the log path in the application container matches the collection configuration. |
The parameter is invalid: uuid=none
Problem description: The LoongCollector (Logtail) log file (/usr/local/ilogtail/ilogtail.LOG) contains the error message The parameter is invalid : uuid=none.
Resolution: On the host, create a file named product_uuid, enter a valid UUID (for example, 169E98C9-ABC0-4A92-B1D2-AA6239C0D261), and then mount this file to the /sys/class/dmi/id/product_uuid directory in the LoongCollector (Logtail) container.
Multiple collection configurations for a single source
By default, to prevent data duplication, Simple Log Service restricts each log source to a single collection configuration:
A text log file can be processed by only one Logtail collection configuration.
For container standard output (stdout):
With the new standard output template, logs can be collected by only one collection configuration by default.
The old standard output template supports collection by multiple configurations by default, without extra configuration.
Log on to the Simple Log Service console and go to the target Project.
In the left-side navigation pane, choose
Logstores and find the target Logstore.Click the
icon next to its name to expand it.Click Logtail Configuration, find the target configuration in the list, and click Manage Logtail Configuration in the Actions column.
On the Logtail configuration page, click Edit and scroll down to the Input Configurations section:
To collect text file logs: Enable Allow File to Be Collected for Multiple Times.
To collect container standard output: Enable Allow Collection by Different Logtail Configurations.
Multi-target distribution configurations
Because multi-target distribution configurations are associated with multiple Logstores, you must manage them from the Project-level configuration page:
Log on to the Simple Log Service console and click the name of the target Project.
On the target Project page, in the left-side navigation pane, click
.NoteThis page lists all collection configurations in the Project, including configurations for Logstores that were accidentally deleted.
Appendix: Native parsing processors
On the Logtail Configuration page, in the Processor Configurations section, you can add processors to structure raw logs. To add a processor to an existing collection configuration:
In the left-side navigation pane, choose
Logstore, and find the target Logstore.Click the
icon next to its name to expand the Logstore.Click Logtail Configuration. In the configuration list, find the target Logtail Configuration and click Manage Logtail Configuration in the Actions column.
On the Logtail Configuration page, click Edit.
Logstore, and find the target Logstore.
icon next to its name to expand the Logstore.This section describes common processors that cover typical log processing scenarios. For more features, see Extension Processors.
Rules for combining processors (applies to LoongCollector / Logtail 2.0 and later):
You can use Native Processors and Extension Processors independently or combine them as needed.
For better performance and stability, prioritize Native Processors.
If native features cannot meet your business needs, you can add Extension Processors after the configured Native Processors for additional processing.
Order Constraint:
All processors form a processing chain and execute in their configured order. Note: All Native Processors must precede any Extension Processors. After you add an Extension Processor, you can no longer add any Native Processors.
Regex parsing
Extracts log fields with regular expressions and parses logs into key-value pairs. Each field can be queried and analyzed independently.
Example:
Raw log | Result |
| |
Procedure: On the Logtail Configuration page, in the Processor Configurations section, click Add Processor, and select :
Regular Expression: A regular expression for matching logs. You can generate it automatically or enter it manually.
Automatic generation:
Click Automatically Generate Regular Expression.
In the Log Sample section, highlight the log content to extract.
Click Generate Regular Expression.
Paste a correctly formatted log, such as an Apache Combined access log, into the Log Sample field. Then, click Generate Regular Expression to automatically generate the parsing expression.
Manual input: Enter a regular expression based on the log format.
After you complete the configuration, click Validate to test whether the regular expression correctly parses the log content.
Extracted Field: Set the field names (Keys) for the extracted log content (Values).
For information about other parameters, see the description of common configuration parameters in Scenario 2: Structured logs.
Delimiter parsing
Splits log content using a delimiter and parses the content into multiple key-value pairs. This feature supports single-character and multi-character delimiters.
Example:
Raw log | Split a field by the |
| |
Procedure: On the Logtail Configuration page, in the Processor Configurations section, click Add Processor, and select :
Delimiter: Specify the character used to split the log content.
Example: For a CSV file, select Custom and enter a comma (,).
Quote: If a field value contains the delimiter, you must specify a quote character to enclose the field to prevent incorrect splitting.
Extracted Field: Set the field name (Key) for each column in the order they are split. The field names must meet the following requirements:
Can contain only letters, digits, and underscores (_).
Must start with a letter or an underscore (_).
Maximum length: 128 bytes.
For information about other parameters, see the description of common configuration parameters in Scenario 2: Structured logs.
JSON parsing
Parses a JSON log into key-value pairs.
Example:
Raw log | Result |
| |
Procedure: On the Logtail Configuration page, in the Processor Configurations section, click Add Processor, and select :
Original Field: The default value is content. This field stores the raw log content to be parsed.
For information about other parameters, see the description of common configuration parameters in Scenario 2: Structured logs.
Nested JSON parsing
Parses a nested JSON log into key-value pairs by specifying an expansion depth.
Example:
Raw log | Result (depth: 0) | Result (depth: 1) |
| | |
Procedure: On the Logtail Configuration page, in the Processor Configurations section, click Add Processor, and select :
Original Field: The name of the original field to expand, for example,
content.JSON Expansion Depth: Specifies the recursion depth for expansion. A value of
0(the default) expands all nested levels. A value of1prevents any expansion, keeping the original JSON object as the value.Character to Concatenate Expanded Keys: The character used to concatenate field names during JSON expansion. The default is an underscore (
_).Name Prefix of Expanded Keys: The prefix for field names after JSON expansion.
Expand Array: Enable this option to expand an array into key-value pairs with indexes.
Example:
{"k":["a","b"]}is expanded to{"k[0]":"a","k[1]":"b"}.To rename an expanded field, for example, from
prefix_s_key_k1tonew_field_name, you can add a Rename Fields processor to complete the mapping.For information about other parameters, see the description of common configuration parameters in Scenario 2: Structured logs.
JSON array parsing
Use the json_extract function to extract JSON objects from a JSON array.
Example:
Raw log | Result |
| |
Procedure: On the Logtail Configuration page, in the Processor Configurations section, switch the Processing Method to SPL, configure the SPL Statement, and use the json_extract function to extract JSON objects from the JSON array.
Example: Extract elements from the JSON array in the content log field and store the results in the new json1 and json2 fields.
* | extend json1 = json_extract(content, '$[0]'), json2 = json_extract(content, '$[1]')Apache log parsing
Parses log content based on the definitions in an Apache log configuration file and parses it into multiple key-value pairs.
Example:
Raw log | Parsing the Apache |
| |
Procedure: On the Logtail Configuration page, in the Processor Configurations section, click Add Processor, and select :
Log Format: combined
APACHE LogFormat Configuration: The system automatically populates this field based on the selected Log Format.
ImportantVerify that the auto-populated content is identical to the
LogFormatdefined in your server's Apache configuration file, which is typically located at/etc/apache2/apache2.conf.For information about other parameters, see the description of common configuration parameters in Scenario 2: Structured logs.
Data masking
Masks sensitive data in logs.
Example:
Raw log | Result |
| |
Procedure: On the Logtail Configuration page, in the Processor Configurations section, click Add Processor, and select :
Original Field: The original field that contains the log content before parsing.
Data Masking Method:
const: Replaces sensitive content with a specified string.
md5: Replaces sensitive content with its MD5 hash.
Replacement String: If you set Data Masking Method to const, you must enter a string to replace the sensitive content.
Content Expression that Precedes Replaced Content: An expression used to locate sensitive content. Configure this parameter using the RE2 syntax.
Content Expression to Match Replaced Content: The expression for the sensitive content. Configure this parameter using the RE2 syntax.
Time parsing
Parses the time field in the log and sets the parsed result as the log's __time__ field.
Example:
Raw log | Result |
| Parses the time value from the |
Procedure: On the Logtail Configuration page, in the Processor Configurations section, click Add Processor, and select :
Original Field: The original field that contains the log content before parsing.
Time Format: Specify the time format that matches the time value in the log.
Time Zone: Select the time zone of the log's time field. By default, this is the time zone of the machine where LoongCollector (or Logtail) is running.
Appendix: Regular expression limitations (container filtering)
The regular expressions used for container filtering are based on Go's RE2 engine, which has some syntax limitations compared to other engines like PCRE.
1. Differences in named group syntax
Go uses the (?P<name>...) syntax to define a named group and does not support the (?<name>...) syntax from PCRE.
Correct example:
(?P<year>\d{4})Incorrect syntax:
(?<year>\d{4})
2. Unsupported regex features
RE2 does not support the following common but complex regex features:
Lookarounds:
(?=...),(?!...),(?<=...),(?<!...)Conditionals:
(?(condition)true|false)Recursion:
(?R),(?0)Subroutine calls:
(?&name),(?P>name)Atomic groups:
(?>...)
3. Usage recommendations
When debugging regular expressions with a tool like Regex101, select the Golang (RE2) flavor to validate them and ensure compatibility. If you use any unsupported syntax, the plugin cannot parse or match your expression correctly.
Appendix: Container standard output version comparison
To improve storage efficiency and collection consistency, the log metadata format for container standard output has been upgraded. In the new format, all metadata is consolidated under the __tag__ field to optimize storage and standardize the format.
Core advantages of the new version
Significant performance gains
Refactored in C++, the new version delivers a 180% to 300% performance improvement over the previous Go implementation.
It supports native plugins for data processing and multi-threaded parallel processing to fully utilize system resources.
It supports flexible combinations of native and Go plugins to handle complex scenarios.
Enhanced reliability
It supports a log rotation queue for container standard output and unifies the log collection mechanism with the file collection mechanism. This unification ensures high reliability during rapid log rotation.
Lower resource consumption
CPU usage is reduced by 20% to 25%.
Memory usage is reduced by 20% to 25%.
Enhanced O&M consistency
Unified parameter configuration: The configuration parameters for the new container standard output collection plugin are consistent with those for the file collection plugin.
Unified metadata management: The field names and storage location for container metadata are now consistent with the file collection format. As a result, the consumer side only needs one set of processing logic.
Feature comparison of new and old versions
Feature
Previous behavior
New behavior
Storage method
Metadata is embedded in the log content as individual fields.
Metadata is consolidated under the
__tag__field.Storage efficiency
Each log entry contains a full, duplicate set of metadata, which consumes more storage space.
Multiple log entries from the same source can reuse metadata, saving storage costs.
Format consistency
The format is inconsistent with the file collection format.
Field names and storage structure are now consistent with the file collection format, providing a unified experience.
Query access method
You can query metadata fields directly by name, such as
_container_name_.You must access the corresponding key-value pair through the
__tag__object, such as__tag__: _container_name_.Container metadata field mapping
Previous parameter
New parameter
_container_ip_
__tag__:_container_ip_
_container_name_
__tag__:_container_name_
_image_name_
__tag__:_image_name_
_namespace_
__tag__:_namespace_
_pod_name_
__tag__:_pod_name_
_pod_uid_
__tag__:_pod_uid_
In the new version, all metadata fields are stored in the log's tag section in the format
__tag__:<key>instead of being embedded in the log content.Impact on users
Consumer-side adaptation: Because the storage location has changed from "Content" to "Tag", you must adjust your log consumption logic. For example, you must use __tag__ to access the field when you run a query.
SQL query compatibility: SQL queries are automatically backward-compatible, so you do not need to modify existing queries to process logs from both versions.
More information
Global parameters
Input parameters
Processor parameters
|
Parameter |
Description |
|
Log sample |
A sample of the log you want to collect. Use a log sample from your actual use case. The sample helps you configure processing parameters more easily. You can add multiple samples. The total length cannot exceed 1,500 characters. |
|
Multi-line mode |
|
|
Processing mode |
Processors, which includes the Native Processor and the Extended Processor. For more information about processors, see Use native and extended processors. Important
For processor usage limitations, refer to the console prompts.
|
Regions
Log on to the Simple Log Service console. In the project list, click the destination project.
Click the
icon next to the project name to go to the project overview page.In the Basic Information section, view the region name of the current project. The following table maps region names to their corresponding Region IDs.
A region is the geographical location of the physical data center for a cloud service. A Region ID is the unique identifier of a cloud service region.
Region name
Region ID
China (Qingdao)
cn-qingdao
China (Beijing)
cn-beijing
China (Zhangjiakou)
cn-zhangjiakou
China (Hohhot)
cn-huhehaote
China (Ulanqab)
cn-wulanchabu
China (Hangzhou)
cn-hangzhou
China (Shanghai)
cn-shanghai
China (Nanjing - Local Region - Decommissioning)
cn-nanjing
China (Fuzhou - Local Region - Decommissioning)
cn-fuzhou
China (Shenzhen)
cn-shenzhen
China (Heyuan)
cn-heyuan
China (Guangzhou)
cn-guangzhou
Philippines (Manila)
ap-southeast-6
South Korea (Seoul)
ap-northeast-2
Malaysia (Kuala Lumpur)
ap-southeast-3
Japan (Tokyo)
ap-northeast-1
Thailand (Bangkok)
ap-southeast-7
China (Chengdu)
cn-chengdu
Singapore
ap-southeast-1
Indonesia (Jakarta)
ap-southeast-5
China (Hong Kong)
cn-hongkong
Germany (Frankfurt)
eu-central-1
US (Virginia)
us-east-1
US (Silicon Valley)
us-west-1
UK (London)
eu-west-1
UAE (Dubai)
me-east-1
SAU (Riyadh - Partner Region)
me-central-1
icon next to the project name to go to the project overview page.