Continuously collect text logs from hosts

Procedure:

1. On the Logstores page, click to the left of the destination Logstore name to expand it.

2. Next to Import Data, click the icon. In the Quick Data Import dialog box, select a text log ingestion template (such as Single-line Text Log) and click Integrate Now.

   All text log ingestion templates differ only in their parsing plug-ins. The rest of the configuration process is the same and can be modified later.

3. On the Machine Group Configurations page, configure the following parameters:

   • Scenario: Servers

   • Installation Environment: ECS

   • Configure Machine Group: Based on the LoongCollector installation status and machine group configuration of the destination server, select the appropriate option:

     • If LoongCollector is installed and has been added to a machine group, select it from the Source Machine Group list and add it to the Applied Machine Group list. You do not need to create it again.

     • If LoongCollector is not installed, click Create Machine Group:

       The following steps guide you through the automatic installation of LoongCollector and the creation of a machine group.

       1. The system automatically lists ECS instances in the same region as the project. Select one or more instances from which to collect logs.

       2. Click Install and Create as Machine Group. The system automatically installs LoongCollector on the selected ECS instances.

       3. Configure the machine group Name and click OK.

       Note

       If the installation fails or remains in a waiting state, check whether the ECS region is the same as the project region.

     • To add a server with LoongCollector already installed to an existing machine group, see How do I add a server to an existing machine group?

4. Check heartbeat status: Click Next. The Machine Group Heartbeat Status section appears. Check the Heartbeat status. If it is OK, the machine group connection is normal. Click Next to go to the Logtail configuration page.

   If the status is FAIL, it may take some time to establish the initial heartbeat. Wait about two minutes, then refresh the heartbeat status. If it is still FAIL after you refresh, see Machine group heartbeat is FAIL for troubleshooting.

Global Configurations:

• Configuration Name: A custom name for the collection configuration. It must be unique within its project. The name cannot be changed after creation. Naming conventions:

  • Can contain only lowercase letters, digits, hyphens (-), and underscores (_).

  • Must start and end with a lowercase letter or a digit.

Input Configurations:

• Type: Text Log Collection.

• File Path: The path of the log file to collect.

  • Linux: Must start with a forward slash (/). For example, /data/mylogs/**/*.log indicates all files with the .log extension in the /data/mylogs directory.

  • Windows: Must start with a drive letter. For example, C:\Program Files\Intel\**\*.Log.

• Maximum Directory Monitoring Depth: The maximum directory depth that the wildcard character ** in the File Path can match. The default is 0, which means only the current directory is monitored.

Scenario 2: Structured logging

192.168.*.* - - [15/Apr/2025:16:40:00 +0800] "GET /nginx-logo.png HTTP/1.1" 0.000 514 200 368 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.*.* Safari/537.36"

body_bytes_sent: 368
http_referer: -
http_user_agent : Mozi11a/5.0 (Nindows NT 10.0; Win64; x64) AppleMebKit/537.36 (KHTML, like Gecko) Chrome/131.0.x.x Safari/537.36
remote_addr:192.168.*.*
remote_user: -
request_length: 514
request_method: GET
request_time: 0.000
request_uri: /nginx-logo.png
status: 200
time_local: 15/Apr/2025:16:40:00

Procedure: In the Processor Configurations area of the Logtail Configuration page

1. Add a parsing plug-in: Click Add Processor and configure a plug-in, such as Regular Expression Parsing, Delimiter Parsing, or JSON Parsing, based on the actual format. This example uses NGINX log collection. Select Native Processor > Data Parsing (NGINX Mode).

2. NGINX Log Configuration: Copy the complete log_format definition from the Nginx server configuration file (nginx.conf) and paste it into this text box.

   Example:

   log_format main  '$remote_addr - $remote_user [$time_local] "$request" ''$request_time $request_length ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent"';

   Important

   The format definition here must be exactly the same as the format that generates the logs on the server. Otherwise, log parsing fails.

3. General configuration parameters: The following parameters appear in multiple data parsing plug-ins, and their functions and usage are consistent.

   • Source Field: Specifies the source field name to be parsed. The default is content, which is the entire collected log entry.

   • Keep Source Field on Parse Failure: We recommend that you enable this option. If a log cannot be successfully parsed by the plug-in, for example, due to a format mismatch, this option ensures that the original log content is not lost but is fully retained in the specified source field.

   • Keep Source Field on Parse Success: If you select this option, the original log content is retained even if the log is parsed successfully.

Procedure: In Global Configurations > Other Global Configurations > Log Topic Type, select a method for generating topics. The following three types are supported:

• Machine group topic: When a collection configuration is applied to multiple machine groups, LoongCollector automatically uses the name of the server's machine group as the __topic__ field for upload. This is suitable for scenarios where logs are categorized by host.

• Custom: The format is customized://<custom_topic_name>, for example, customized://app-login. This is suitable for static topic scenarios with fixed service identifiers.

• File path extraction: Extract key information from the full path of the log file to dynamically mark the log source. This is suitable for situations where multiple users or applications share the same log filename but have different paths. When multiple users or services write logs to different top-level directories but the sub-paths and filenames are the same, the source cannot be distinguished by filename alone. For example:

  /data/logs
  ├── userA
  │   └── serviceA
  │       └── service.log
  ├── userB
  │   └── serviceA
  │       └── service.log
  └── userC
      └── serviceA
          └── service.log

  In this case, you can configure File path extraction and use a regular expression to extract key information from the full path. The matched result is then uploaded to the Logstore as the topic.

  File path extraction rule: Based on regular expression capturing groups

  When you configure a regular expression, the system automatically determines the output field format based on the number and naming of the capturing groups. The rules are as follows:

  In the regular expression for the file path, you need to escape the forward slash (/).

  Capturing group type	Scenario	Generated field	Regex example	Matching path example	Generated field example
  Single capturing group (only one (.*?))	Only one dimension is needed to distinguish the source (such as username, environment)	Generates the __topic__ field	\/logs\/(.*?)\/app\.log	/logs/userA/app.log	__topic__: userA
  Multiple non-named capturing groups (multiple (.*?))	Multiple dimensions are needed to distinguish the source, but no semantic tags are required	Generates a tag field __tag__:__topic_{i}__, where {i} is the ordinal number of the capturing group	\/logs\/(.*?)\/(.*?)\/app\.log	/logs/userA/svcA/app.log	__tag__:__topic_1__userA __tag__:__topic_2__svcA
  Multiple named capturing groups (using (?P<name>.*?))	Multiple dimensions are needed to distinguish the source, and you want the field meanings to be clear for easy querying and analysis	Generates a tag field __tag__:{name}	\/logs\/(?P<user>.*?)\/(?P<service>.*?)\/app\.log	/logs/userA/svcA/app.log	__tag__:user:userA; __tag__:service:svcA

By default, all logs are sent to the current Logstore, and the compression method is lz4. To distribute logs from the same source to different Logstores, you can configure the following settings:

By default, to avoid data duplication, Simple Log Service restricts a text log file to be collected by only one Logtail configuration. To enable the same log file to be collected by multiple configurations simultaneously, you need to manually enable the feature that allows a file to be collected multiple times.

Procedure:

1. Log on to the Simple Log Service console and go to the destination project.

2. In the navigation pane on the left, choose Logstores and find the destination Logstore.

3. Click to the left of its name to expand the Logstore.

4. Click Logtail Configuration. In the configuration list, find the destination Logtail configuration and click Manage Logtail Configuration in the Actions column.

5. On the Logtail configuration page, click Edit:

   • In Input Configurations > Other Input Configurations, enable Allow File to Be Collected for Multiple Times.

6. After the configuration is complete, click OK.

Extract log fields using regular expressions and parse the logs into key-value pairs. Each field can be independently queried and analyzed.

127.0.0.1 - - [16/Aug/2024:14:37:52 +0800] "GET /wp-admin/admin-ajax.php?action=rest-nonce HTTP/1.1" 200 41 "http://www.example.com/wp-admin/post-new.php?post_type=page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0"

body_bytes_sent: 41
http_referer: http://www.example.com/wp-admin/post-new.php?post_type=page
http_user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; ×64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0
remote_addr: 127.0.0.1
remote_user: -
request_method: GET
request_protocol: HTTP/1.1
request_uri: /wp-admin/admin-ajax.php?action=rest-nonce
status: 200
time_local: 16/Aug/2024:14:37:52 +0800

Procedure: On the Logtail Configuration page, in the Processor Configurations area, click Add Processor and select Native Processor > Data Parsing (Regex Mode):

• Regular Expression: Used to match logs. You can generate it automatically or enter it manually:

  • Automatic generation:

    • Click Generate Regex Automatically.

    • In the Log Sample, select the log content to extract.

    • Click Generate Regex.

      

  • Manual input: Enter Regex Manually based on the log format.

  After the configuration is complete, click Validate to test whether the regular expression can correctly parse the log content.

• Extracted Field: Set the corresponding field name (Key) for the extracted log content (Value).

Structure log content using a separator, which parses it into multiple key-value pairs. Supports single-character and multi-character separators.

05/May/2025:13:30:28,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java

ip:10.10.*.*
request:POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1
size:18204
status:200
time:05/May/2025:13:30:28
user_agent:aliyun-sdk-java

Procedure: On the Logtail Configuration page, in the Processor Configurations area, click Add Processor and select Native Processor > Data Parsing (Delimiter Mode):

• Delimiter: Specify the character used to split the log content.

  Example: For a CSV file, select Custom and enter a comma (,).

• Quote: When a field value contains the separator, you need to specify a quote to enclose the field to avoid incorrect splitting.

• Extracted Field: Set the corresponding field name (Key) for each column in order of separation. The rules are as follows:

  • Field names can contain only letters, digits, and underscores (_).

  • Must start with a letter or an underscore (_).

  • Maximum length: 128 bytes.

Structures an object-type JSON log, which parses it into key-value pairs.

{"url": "POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek********&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1", "ip": "10.200.98.220", "user-agent": "aliyun-sdk-java", "request": {"status": "200", "latency": "18204"}, "time": "05/Jan/2025:13:30:28"}

ip: 10.200.98.220
request: {"status": "200", "latency" : "18204" }
time: 05/Jan/2025:13:30:28
url: POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek******&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1
user-agent:aliyun-sdk-java

Procedure: On the Logtail Configuration page, in the Processor Configurations area, click Add Processor and select Native Processor > Data Parsing (JSON Mode):

• Original Field: The default value is content. This field is used to store the raw log content to be parsed.

Parses nested JSON logs into key-value pairs by specifying the expansion depth.

{"s_key":{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}}

0_s_key_k1_k2_k3_k41:41
0_s_key_k1_k2_k3_k4_k51:51
0_s_key_k1_k2_k3_k4_k52:52

1_s_key:{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}

Procedure: On the Logtail Configuration page, in the Processor Configurations area, click Add Processor and select Extended Processor > Expand JSON Field:

• Original Field: The name of the source field to be expanded, for example, content.

• JSON Expansion Depth: The expansion level of the JSON object. 0 means fully expanded (default), 1 means the current level, and so on.

• Character to Concatenate Expanded Keys: The separator for field names during JSON expansion. The default is an underscore _.

• Name Prefix of Expanded Keys: Specify the prefix for field names after JSON expansion.

• Expand Array: Enable this to expand an array into key-value pairs with indexes.

  Example: {"k":["a","b"]} expands to {"k[0]":"a","k[1]":"b"}.

  To rename the expanded fields, for example, from prefix_s_key_k1 to new_field_name, you can add a Rename Fields plug-in afterward to complete the mapping.

Use the json_extract function to extract JSON objects from a JSON array.

Example:

[{"key1":"value1"},{"key2":"value2"}]

json1:{"key1":"value1"}
json2:{"key2":"value2"}

Procedure: On the Logtail Configuration page, in the Processor Configurations area, switch the Processing Method to SPL, configure the SPL Statement, and use the json_extract function to extract JSON objects from the JSON array.

Example: Extract elements from the JSON array in the log field content and store the results in new fields json1 and json2.

* | extend json1 = json_extract(content, '$[0]'), json2 = json_extract(content, '$[1]')

Structures the log content based on the definition in the Apache log configuration file, which parses it into multiple key-value pairs.

1 192.168.1.10 - - [08/May/2024:15:30:28 +0800] "GET /index.html HTTP/1.1" 200 1234 "https://www.example.com/referrer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36"

http_referer:https://www.example.com/referrer
http_user_agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36
remote_addr:192.168.1.10
remote_ident:-
remote_user:-
request_method:GET
request_protocol:HTTP/1.1
request_uri:/index.html
response_size_bytes:1234
status:200
time_local:[08/May/2024:15:30:28 +0800]

Procedure: On the Logtail Configuration page, in the Processor Configurations area, click Add Processor and select Native Processor > Data Parsing (Apache Mode):

• Log Format: combined.

• APACHE LogFormat Configuration: The system automatically fills in the configuration based on the Log Format.

  Important

  Be sure to verify the auto-filled content to ensure it is exactly the same as the LogFormat defined in your server's Apache configuration file, which is usually located at /etc/apache2/apache2.conf.

Structures the log content based on the IIS log format definition, which parses it into multiple key-value pairs.

Comparison example:

#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken

c-ip: cs-username
cs-bytes: sc-substatus
cs-method: cs-method
cs-uri-query: cs-uri-query
cs-uri-stem: cs-uri-stem
cs-username: s-port
date: #Fields:
s-computername: s-sitename
s-ip: s-ip
s-sitename: time
sc-bytes: sc-status
sc-status: c-ip
sc-win32-status: cs (User-Agent)
time: date
time-taken: sc-win32-status

Procedure: On the Logtail Configuration page, in the Processor Configurations area, click Add Processor and select Native Processor > Data Parsing (IIS Mode):

• Log Format: Select the log format used by your IIS server logs.

  • IIS: IIS log file format.

  • NCSA: NCSA Common Log Format.

  • W3C: W3C Extended Log File Format.

• IIS Configuration Fields: When you select IIS or NCSA, Simple Log Service sets the IIS configuration field by default. When you select W3C, set it to the content of the logExtFileFlags parameter in your IIS configuration file. For example:

  logExtFileFlags="Date, Time, ClientIP, UserName, SiteName, ComputerName, ServerIP, Method, UriStem, UriQuery, HttpStatus, Win32Status, BytesSent, BytesRecv, TimeTaken, ServerPort, UserAgent, Cookie, Referer, ProtocolVersion, Host, HttpSubStatus"

Mask sensitive data in logs.

Example:

[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]

[{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

Procedure: On the Logtail Configuration page, in the Processor Configurations area, click Add Processor and select Native Processor > Data Masking:

• Original Field: The source field that contains the log content before parsing.

• Data Masking Method:

  • const: Replaces sensitive content with the specified string.

  • md5: Replaces sensitive content with its corresponding MD5 hash.

• Replacement String: When you select const for Data Masking Method, you need to enter a string to replace the sensitive content.

• Content Expression that Precedes Replaced Content: Used to find sensitive content. Configure using RE2 syntax.

• Content Expression to Match Replaced Content: The expression for the sensitive content. Configure using RE2 syntax.

Parses the time field in the log and sets the parsing result as the log's __time__ field.

Example:

{"level":"INFO","timestamp":"2025-09-23T19:11:47+0800","cluster":"yilu-cluster-0728","message":"User logged in successfully","userId":"user-123"}

Procedure: On the Logtail Configuration page, in the Processor Configurations area, click Add Processor and select Native Processor > Time Parsing:

• Original Field: The source field that contains the log content before parsing.

• Time Format: Set the corresponding time format based on the time content in the log.

• Time Zone: Select the time zone of the log time field. By default, the machine time zone is used, which is the time zone of the environment where the LoongCollector (Logtail) process is running.