Collect logs in simple mode

Procedure

1. Log on to the Simple Log Service console.

2. In the Import Data section, select Multi-line text log.

   This topic uses a multi-line text log as an example. If you want to collect a single-line text log, select Single-line text log.

3. Select the destination project and Logstore, and then click Next.

4. Create a machine group.

   • If you have a machine group, click Use Existing Machine Groups.

   • If you do not have a machine group, follow these steps to create one. This example uses an ECS instance.

     1. On the ECS Instance tab, select the target ECS instance and click Create.

        For more information, see Install Logtail on ECS instances.

        Important

        If your server is an ECS instance from a different Alibaba Cloud account, a server from another cloud provider, or a server in your own data center, you must install Logtail manually. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server.

        After you manually install Logtail, you must configure a user identifier on the server. For more information, see Configure a user identifier.

     2. After the installation is complete, click Complete Installation.

     3. On the Create Machine Group page, enter a Name and click Next.

        Log Service supports IP address-based machine groups and custom identifier-based machine groups. For more information about the parameters, see Create an IP address-based machine group and Create a custom identifier-based machine group.

5. Select the target machine group from Source Machine Group, move it to Applied Machine Group, and then click Next.

   Important

   If you apply a machine group immediately after creating it, the heartbeat status may appear as FAIL because the connection is not yet established. You can click Automatic Retry. If the issue persists, see Troubleshoot Logtail machine group heartbeat failures for troubleshooting steps.

6. Create a Logtail configuration and click Next.

   Parameter	Description
   Configuration name	The name of the Logtail configuration. It must be unique within the project. You cannot change the name after the configuration is created. You can also click Import Other Configuration to use an existing Logtail configuration.
   Log path	Specify the directory and file name of your logs on the server. A log path for a Linux system must start with a forward slash (/), such as /apsara/nuwa/.../app.Log. If the log path is on a Windows system, it must start with a drive letter, for example, C:\Program Files\Intel\...\*.Log. Directory and file names support both full names and wildcards. For more information on file name patterns, see Wildcard matching. Logtail recursively searches the specified directory path to find all matching files. For example: /apsara/nuwa/**/*.log represents the files that end with .log in the /apsara/nuwa directory and its subdirectories. /var/logs/app_*/**/*.log refers to all files with the .log extension in all directories that match the app_* pattern (including their recursive subdirectories) under the /var/logs directory. /var/log/nginx/**/access* specifies the files that start with access in the /var/log/nginx directory and its recursive subdirectories. Note Log path wildcards support only the asterisk (*) and the question mark (?). An asterisk (*) matches any sequence of characters. A question mark (?) matches any single character.
   Blacklist	Enable Blacklist to exclude specific directories or files from collection. Both full paths and wildcards are supported. For example: If you select Filter by Directory and set the path to /home/admin/dir1, all content in the /home/admin/dir1 directory is ignored during collection. Select Filter by Directory and set the path to /home/admin/dir* to ignore the content of all subdirectories that start with dir under the /home/admin/ directory during collection. Select Filter by Directory and set the path to /home/admin/*/dir. This configuration ignores all content in any second-level subdirectory named dir within the /home/admin/ directory during collection. For example, the content in the /home/admin/a/dir directory is ignored, and the content in the /home/admin/a/b/dir directory is collected. Select Filter by File, and set the path to /home/admin/private*.log. This setting ignores all files in the /home/admin/ directory that start with private and end with .log during collection. Select Filter by File and set the path to /home/admin/private*/*_inner.log to ignore files that end with _inner.log and are located in directories that start with private under the /home/admin/ directory during collection. For example, the file /home/admin/private/app_inner.log is ignored, and the file /home/admin/private/app.log is collected. Note Directory wildcards support only the asterisk (*) and the question mark (?). An asterisk (*) matches any sequence of characters. A question mark (?) matches any single character. If you use wildcards in the Log Path and need to exclude a subdirectory, you must specify its full path in the blacklist for the exclusion to apply correctly. For example, if you set the log path to /home/admin/app*/log/*.log but want to filter out all subdirectories under the /home/admin/app1* directory, you need to configure a blacklist by selecting Filter by Directory and setting the path to /home/admin/app1*/** . If you set the path to /home/admin/app1*, the blacklist will not take effect. Matching against a blacklist consumes computing resources. We recommend keeping the number of blacklist entries under 10. A directory path cannot end with a forward slash (/). For example, if you set the path to /home/admin/dir1/, the directory blacklist will not take effect.
   Docker file	If the log file is in a Docker container, enable Docker file and configure the internal path and container tags. Logtail automatically monitors container creation and destruction, filters containers based on tags, and collects logs from the specified containers. For more information, see Use the console to collect text logs from containers in DaemonSet mode.
   Mode	The collection mode. The default is simple mode - multi-line. You can select a different mode.
   Log sample	Provide a log sample from your actual use case. Log Service uses this sample to automatically generate the first-line regular expression. For example: [2020-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened at TestPrintStackTrace.f(TestPrintStackTrace.java:3) at TestPrintStackTrace.g(TestPrintStackTrace.java:7) at TestPrintStackTrace.main(TestPrintStackTrace.java:16) This parameter is not required when you collect single-line text logs in simple mode.
   First-line regular expression	Logtail uses a first-line regular expression to identify the start of a log entry. Lines that do not match the expression are appended to the previous entry. Log Service can automatically generate the expression or you can enter it manually. Automatically generate After you enter a log sample, click Automatically Generate to generate the first-line regular expression. Manually enter After you enter a log sample, click Manually Enter Regular Expression to enter your own expression. After you enter the expression, click Validate to verify it. For more information, see How do I test a regular expression?. This parameter is not required when you collect single-line text logs in simple mode.
   Drop logs that fail to parse	Specifies whether to drop logs that fail to parse. If you enable drop logs that fail to parse, logs that fail to parse are not uploaded to Log Service. If you turn off the Drop logs that fail to parse switch, the raw log is uploaded to Log Service as the value of the raw_log field when log parsing fails.
   Maximum monitoring directory depth	The maximum depth of subdirectories to monitor within the specified log directory. The value can range from 0 to 1,000. A value of 0 means only the top-level directory is monitored.

   Configure advanced settings based on your needs. If you have no special requirements, we recommend that you keep the default settings.

   Parameter	Description
   Enable plugin processing	If your logs are too complex or unstructured for standard parsing modes such as NGINX, full regular expression, or JSON, you can enable Enable Plug-in Processing and add a Logtail plugin to process the logs. Log Service generates a default plugin configuration based on your selected collection mode. You can add more plugins by adding multiple processors. For more information, see Overview of Logtail plugins for data processing. Important If you enable Enable Plug-in Processing, features such as Upload raw logs, Timezone, Drop logs that fail to parse, and Filter configuration become unavailable.
   Upload raw logs	If you turn on the Upload Raw Log switch, the raw log will be uploaded to Log Service as the value of the raw field, along with the parsed log.
   Topic generation mode	Specifies how to generate a topic for your logs. For more information, see Log topics. None - Do Not Generate Topic: This is the default option. The topic is set to an empty string. You do not need to specify a topic when you query logs. Machine Group Topic Attributes: Uses the topic attribute of the machine group. This is useful for distinguishing logs generated by different servers. File Path Regular Expression: Uses a regular expression to extract a topic from the log file path. You must specify the expression in the Custom regex field. This is useful for distinguishing logs from different users or instances.
   Log file encoding	The encoding format of the log files. Valid values are utf8 and gbk.
   Timezone	The timezone for the log's timestamp during collection. Machine timezone: The default value. Uses the timezone of the host or container where Logtail is running. Custom: Allows you to manually select a timezone.
   Timeout	If a log file is not updated within the specified period, Logtail considers the monitoring to have timed out. Never Timeout: Monitors all log files continuously without timing out. 30-Minute Timeout: If a log file is not updated for 30 minutes, Logtail stops monitoring the file. When you select Expire in 30 Minutes, you must also set Maximum timeout directory depth to a value from 1 to 3.
   Filter configuration	Only logs that meet the filter conditions are collected. The filter conditions include the key field, which specifies the name of the target field, and the Regex field, which specifies the regular expression for filtering. A logical AND is applied between these conditions. Important In simple mode, only the content field is generated. If you want to set a filter condition, you can only set the Key to content. The regular expression in this setting supports only full-text matching, not partial keyword matching. The following are examples: Collecting matching logs. Set Key to level and Regex to WARNING|ERROR. This collects only logs where the level field is WARNING or ERROR. Set Key to content and Regex to .05\/Jan\/2023., which collects only logs where the content field contains 05/Jan/2023. Excluding logs. Set Key to level and Regex to ^(?!.*(INFO|DEBUG)).* to exclude logs where the level field contains INFO or DEBUG. Set Key to level and Regex to ^(?!(INFO|DEBUG)$).*. This means that logs with a level field value of INFO or DEBUG are not collected. Set Key to url and Regex to .*^(?!.*(healthcheck)).* to exclude logs in which the url field contains healthcheck. For more information about how to configure regular expressions, see Getting started with regular expressions.
   Initial collection size	First Collection Size determines the starting point for collecting data from a new file. The default size is 1,024 KB. If the file is smaller than 1,024 KB, collection starts from the beginning of the file. If the file is larger than 1,024 KB, collection starts 1,024 KB from the end of the file. You can change the First Collection Size. The value can range from 0 to 10,485,760. Unit: KB.
   Advanced configuration	Specifies advanced Logtail settings. For more information, see Advanced parameters. For example, to use this Logtail configuration to collect logs from a file that is already matched by another configuration and to specify the aggregation interval, you can add the following configuration: { "force_multiconfig": true, "batch_send_interval": 3 }

   Click Next to create the Logtail configuration. Log Service then begins collecting logs.

   Note

   • The Logtail configuration may take up to 3 minutes to take effect.

   • If you encounter collection errors, see How do I view Logtail collection errors? and Common collection errors.

7. Preview the data, create indexes, and then click Next.

   By default, Log Service enables full-text indexing. You can also manually create field indexes based on the collected logs, or click Automatic Index Generation to have Log Service create them automatically. For more information, see Create indexes.

   Important

   To query and analyze logs, you must enable either full-text indexing or a field index. If you enable both, queries use the field indexes.

8. Click Query Log to go to the Query & Analysis page for the Logstore.

   You may need to wait about one minute for the indexes to become effective. You can then view the collected logs on the Raw Logs tab. For more information, see Quick start for log query and analysis.

Troubleshooting

If no data appears on the preview or query page after you configure Logtail to collect logs, see Troubleshoot Logtail collection failures.