Collect text logs from servers

Procedure:

1. On the Logstores page, click the icon before the target logstore name to expand it.

2. Click the icon next to Data Collection, and in the Quick Data Import dialog box, select a text log collection template (such as Single Line - Text Log) and click Integrate Now.

   All text log templates share the same configuration and differ only in parsing plugins. You can modify the configuration later.

3. On the Machine Group Configurations page, configure the following parameters:

   • Scenario: Servers

   • Installation Environment: ECS

   • Select Machine Group: Based on the LoongCollector installation status and machine group configuration of the target server, select the corresponding operation:

     • If LoongCollector is already installed and added to a machine group, select the machine group from Source Machine Group and move it to Applied Server Groups.

     • If LoongCollector is not installed, click Create Machine Group:

       Follow these steps to install LoongCollector automatically and create a machine group.

       1. The system lists ECS instances in the same region as the project. Select the instances to collect logs from.

       2. Click Install and Create Machine Group. The system automatically installs LoongCollector on the selected ECS instances.

       3. Configure the machine group Name and click OK.

       Note

       If the installation fails or remains in a waiting state, check whether the ECS region is the same as the project region.

     • To add a server with LoongCollector already installed to an existing machine group, see the FAQ How do I add a server to an existing machine group?

4. Check the heartbeat status: Click Next. The Machine Group Heartbeat section appears. Check the Heartbeat status. If it is OK, the machine group connection is normal. Click Next to go to the Logtail configuration page.

   If the status is FAIL, wait about two minutes and refresh. If still FAIL, see Machine group heartbeat is FAIL for further troubleshooting.

Global Configurations:

• Configuration Name: A custom name for the collection configuration. This name must be unique within the project and cannot be changed after it is created. Naming conventions:

  • Can contain only lowercase letters, digits, hyphens (-), and underscores (_).

  • Must start and end with a lowercase letter or a digit.

Input Configurations:

• Type: Text Log Collection.

• File Path: The log collection path.

  • Linux: The path must start with a forward slash (/). For example, /data/mylogs/**/*.log indicates all files with the .log extension in the /data/mylogs directory and its subdirectories.

  • Windows: The path must start with a drive letter. For example, C:\Program Files\Intel\**\*.Log.

• Maximum Directory Monitoring Depth: The maximum depth of directories matched by the ** wildcard character in the File Path. The default value is 0, which indicates that only the current directory is monitored.

Use case 2: Structured logs

192.168.*.* - - [15/Apr/2025:16:40:00 +0800] "GET /nginx-logo.png HTTP/1.1" 0.000 514 200 368 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.*.* Safari/537.36"

body_bytes_sent: 368
http_referer: -
http_user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.x.x Safari/537.36
remote_addr:192.168.*.*
remote_user: -
request_length: 514
request_method: GET
request_time: 0.000
request_uri: /nginx-logo.png
status: 200
time_local: 15/Apr/2025:16:40:00

Procedure: In the Processor Configurations section of the Logtail Configuration page

1. Add a parsing plugin: Click Add Processor and configure a plugin such as regular expression parsing, separator parsing, or JSON parsing based on the actual format. This example uses NGINX log collection and selects Native Processor > Data Parsing (NGINX Mode).

2. NGINX Log Configuration: Copy the log_format definition from the NGINX server configuration file (nginx.conf) and paste it into this text box.

   Example:

   log_format main  '$remote_addr - $remote_user [$time_local] "$request" ''$request_time $request_length ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent"';

   Important

   The format definition here must be exactly the same as the format that generates the logs on the server. Otherwise, log parsing fails.

3. Description of common configuration parameters: The following parameters appear in multiple data parsing plugins and have a unified function and usage.

   • Original Field: The source field to parse. Default: content (the entire collected log entry).

   • Retain Original Field if Parsing Fails: Preserves the original log content when parsing fails (for example, due to a format mismatch).

   • Retain Original Field if Parsing Succeeds: Retains the original log content even after successful parsing.

Procedure: Global Configurations > Other Global Configurations > Log Topic Type: Select a method for generating topics. The following three types are supported:

• Machine Group Topic: When a collection configuration is applied to multiple machine groups, LoongCollector uses the server's machine group name as the __topic__ field. Suitable for dividing logs by host.

• Custom: Uses the format customized://<custom_topic_name>, such as customized://app-login. This format is suitable for static topic use cases with fixed business identifiers.

• File Path Extraction: Extract key information from the log file path to dynamically mark the log source. This is suitable when multiple users or applications share the same log filename but differ in path:

  /data/logs
  ├── userA
  │   └── serviceA
  │       └── service.log
  ├── userB
  │   └── serviceA
  │       └── service.log
  └── userC
      └── serviceA
          └── service.log

  Configure File Path Extraction and use a regular expression to extract key information from the full path. The matched result is then uploaded to the logstore as the topic.

  File path extraction rule: Based on regular expression capturing groups

  When you configure a regular expression, the system automatically determines the output field format based on the number and naming of capturing groups. The rules are as follows:

  In the regular expression for a file path, you must escape the forward slash (/).

  Capturing group type	Use case	Generated field	Regex example	Matching path example	Generated field example
  Single capturing group (only one (.*?))	Only one dimension is needed to distinguish the source (such as username or environment)	Generates the __topic__ field	\/logs\/(.*?)\/app\.log	/logs/userA/app.log	__topic__: userA
  Multiple capturing groups - unnamed (multiple (.*?))	Multiple dimensions are needed to distinguish the source, but no semantic tags are required	Generates a tag field __tag__:__topic_{i}__, where {i} is the ordinal number of the capturing group	\/logs\/(.*?)\/(.*?)\/app\.log	/logs/userA/svcA/app.log	__tag__:__topic_1__userA __tag__:__topic_2__svcA
  Multiple capturing groups - named (using (?P<name>.*？)	Multiple dimensions are needed to distinguish the source, and the field meanings should be clear for easy querying and analysis	Generates a tag field __tag__:{name}	\/logs\/(?P<user>.*?)\/(?P<service>.*?)\/app\.log	/logs/userA/svcA/app.log	__tag__:user:userA; __tag__:service:svcA

By default, all logs are sent to the current logstore with lz4 compression. To send logs from the same source to different logstores:

By default, each text log file can match only one Logtail configuration. To allow multiple configurations to collect the same file, enable multi-collection.

Procedure:

1. Log on to the Simple Log Service console and go to the target project.

2. In the navigation pane on the left, choose Logstores and find the target logstore.

3. Click the icon before its name to expand the logstore.

4. Click Logtail Configuration. In the configuration list, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.

5. On the Logtail configuration page, click Edit:

   • In Input Configurations > Other Input Configurations, enable Allow File to Be Collected for Multiple Times.

6. After the configuration is complete, click Save.

Extract log fields using a regular expression and parse the log into key-value pairs. Each field can be independently queried and analyzed.

127.0.0.1 - - [16/Aug/2024:14:37:52 +0800] "GET /wp-admin/admin-ajax.php?action=rest-nonce HTTP/1.1" 200 41 "http://www.example.com/wp-admin/post-new.php?post_type=page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0"

body_bytes_sent: 41
http_referer: http://www.example.com/wp-admin/post-new.php?post_type=page
http_user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; ×64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0
remote_addr: 127.0.0.1
remote_user: -
request_method: GET
request_protocol: HTTP/1.1
request_uri: /wp-admin/admin-ajax.php?action=rest-nonce
status: 200
time_local: 16/Aug/2024:14:37:52 +0800

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (Regex Mode):

• Regular Expression: The expression used to match logs. Generate it automatically or enter it manually:

  • Automatic generation:

    • Click Generate.

    • In the Log Sample, select the log content to extract.

    • Click Generate Regular Expression.

      

  • Manual entry: Manually Enter Regular Expression based on the log format.

  After configuration, click Validate to test whether the regular expression can correctly parse the log content.

• Extracted Field: The field name (Key) that corresponds to the extracted log content (Value).

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Structure log content using a separator to parse it into multiple key-value pairs. Both single-character and multi-character separators are supported.

05/May/2025:13:30:28,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java

ip:10.10.*.*
request:POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1
size:18204
status:200
time:05/May/2025:13:30:28
user_agent:aliyun-sdk-java

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (Delimiter Mode):

• Delimiter: Specifies the character used to split log content.

  Example: For a CSV file, select Custom and enter a comma (,).

• Quote: If a field value contains the separator, you must enclose the field value in quotes to prevent incorrect splitting.

• Extracted Field: Specify the field name (Key) for each column in the order that they appear. The rules are as follows:

  • Field names can contain only letters, digits, and underscores (_).

  • Must start with a letter or an underscore (_).

  • Maximum length: 128 bytes.

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Structure an Object-type JSON log by parsing it into key-value pairs.

{"url": "POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek********&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1", "ip": "10.200.98.220", "user-agent": "aliyun-sdk-java", "request": {"status": "200", "latency": "18204"}, "time": "05/Jan/2025:13:30:28"}

ip: 10.200.98.220
request: {"status": "200", "latency" : "18204" }
time: 05/Jan/2025:13:30:28
url: POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek******&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1
user-agent:aliyun-sdk-java

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (JSON Mode):

• Original Field: The field that contains the raw log to be parsed. The default value is content.

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Parse a nested JSON log into key-value pairs by specifying the expansion depth.

{"s_key":{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}}

0_s_key_k1_k2_k3_k41:41
0_s_key_k1_k2_k3_k4_k51:51
0_s_key_k1_k2_k3_k4_k52:52

1_s_key:{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Extended Processor > Expand JSON Field:

• Original Field: Specifies the name of the source field to expand, such as content.

• JSON Expansion Depth: The expansion depth of the JSON object, where 0 (the default) indicates full expansion, 1 indicates expansion of the current level, and so on.

• Character to Concatenate Expanded Keys: The separator for field names when a JSON object is expanded. The default value is an underscore (_).

• Name Prefix of Expanded Keys: The prefix for field names after JSON expansion.

• Expand Array: Expands an array into key-value pairs with indexes.

  Example: {"k":["a","b"]} is expanded to {"k[0]":"a","k[1]":"b"}.

  To rename the expanded fields (for example, from prefix_s_key_k1 to new_field_name), add a rename fields plugin afterward to complete the mapping.

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Use the json_extract function to extract JSON objects from a JSON array.

Example:

[{"key1":"value1"},{"key2":"value2"}]

json1:{"key1":"value1"}
json2:{"key2":"value2"}

Procedure: In the Processor Configurations section of the Logtail Configuration page, switch the Processing Mode to SPL, configure the SPL Statement, and use the json_extract function to extract JSON objects from the JSON array.

Example: Extract elements from the JSON array in the log field content and store the results in new fields json1 and json2.

* | extend json1 = json_extract(content, '$[0]'), json2 = json_extract(content, '$[1]')

Structure the log content into multiple key-value pairs based on the definition in the Apache log configuration file.

1 192.168.1.10 - - [08/May/2024:15:30:28 +0800] "GET /index.html HTTP/1.1" 200 1234 "https://www.example.com/referrer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36"

http_referer:https://www.example.com/referrer
http_user_agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36
remote_addr:192.168.1.10
remote_ident:-
remote_user:-
request_method:GET
request_protocol:HTTP/1.1
request_uri:/index.html
response_size_bytes:1234
status:200
time_local:[08/May/2024:15:30:28 +0800]

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (Apache Mode):

• The Log Format is combined.

• The APACHE LogFormat Configuration are automatically populated based on the Log Format.

  Important

  Make sure to verify the auto-filled content to ensure it is exactly the same as the LogFormat defined in your server's Apache configuration file (usually located at /etc/apache2/apache2.conf).

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Structure the log content into multiple key-value pairs based on the IIS log format definition.

Comparison example:

#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken

c-ip: cs-username
cs-bytes: sc-substatus
cs-method: cs-method
cs-uri-query: cs-uri-query
cs-uri-stem: cs-uri-stem
cs-username: s-port
date: #Fields:
s-computername: s-sitename
s-ip: s-ip
s-sitename: time
sc-bytes: sc-status
sc-status: c-ip
sc-win32-status: cs (User-Agent)
time: date
time-taken: sc-win32-status

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Parsing (IIS Mode):

• Log Format: Select the log format for your IIS server.

  • IIS: The log file format for Microsoft Internet Information Services.

  • NCSA: Common Log Format.

  • W3C refers to the W3C Extended Log File Format.

• IIS Configuration Fields: When you select IIS or NCSA, SLS uses the default IIS configuration fields. When you select W3C, you must set the fields to the value of the logExtFileFlags parameter in your IIS configuration file. For example:

  logExtFileFlags="Date, Time, ClientIP, UserName, SiteName, ComputerName, ServerIP, Method, UriStem, UriQuery, HttpStatus, Win32Status, BytesSent, BytesRecv, TimeTaken, ServerPort, UserAgent, Cookie, Referer, ProtocolVersion, Host, HttpSubStatus"

• For other parameters, see the description of common configuration parameters in Use case 2: Structured logs.

Mask sensitive data in logs.

Example:

[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]

[{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Data Masking:

• Original Field: The field that contains the log content before parsing.

• Data Masking Method:

  • const: Replaces sensitive content with a constant string.

  • md5: Replaces sensitive content with its MD5 hash.

• Replacement String: If Data Masking Method is set to const, enter a string to replace the sensitive content.

• Content Expression that Precedes Replaced Content: The expression used to find sensitive content, which is configured using RE2 syntax.

• Content Expression to Match Replaced Content: The regular expression used to match sensitive content. The expression must be written in RE2 syntax.

Parse the time field in the log and set the parsing result as the log's __time__ field.

Example:

{"level":"INFO","timestamp":"2025-09-23T19:11:47+0800","cluster":"yilu-cluster-0728","message":"User logged in successfully","userId":"user-123"}

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and select Native Processor > Time Parsing:

• Original Field: The field that contains the log content before parsing.

• Time Format: Set the time format that corresponds to the timestamps in the log.

• Time Zone: Select the time zone for the log time field. By default, this is the time zone of the environment where the LoongCollector (Logtail) process is running.