Associate machine groups with collection configurations

Machine group

A machine group is a logical group of servers managed as a project resource. Servers connect to machine groups through heartbeats. One configuration can apply to multiple groups, and one group can bind multiple configurations.

SLS supports two types of machine groups:

IP host group

Add server IP addresses directly.

• Simpler to set up.

• An IP address conflict or change can cause heartbeat failures, affecting data collection.

User-defined machine group

Configure a custom identifier and add it to an identifier file on the server. Each identifier occupies one line; a server can have multiple identifiers.

• More complex to configure, but avoids IP address conflicts that can cause collection failures in custom network environments like a VPC.

• Supports automatic scaling. Configure new servers with the same custom identifier, and Simple Log Service automatically adds them to the machine group. To stop collecting logs from a server, delete its identifier file and the server is automatically removed.

• Create a separate machine group for each module to classify log data. For example, a website with an HTTP request module, a logic module, and a storage module can use custom identifiers such as http_module, logic_module, and store_module, respectively.

Create a machine group

Servers require LoongCollector for heartbeat communication with machine groups. Create the machine group when you Install LoongCollector or complete installation and configuration.

Step 1 (Optional): Configure user identifier

1. Log on to the Log Service console with your Alibaba Cloud account (main account). Hover over the profile icon in the upper-right corner and copy the account ID. If you log on as a RAM user, copy the Alibaba Cloud Account ID.

2. Log on to the target server and create an Alibaba Cloud account ID file.

   Linux

   • In the /etc/ilogtail/users directory, create a file named after the Alibaba Cloud account ID.

     touch /etc/ilogtail/users/{Alibaba Cloud account ID}

     Important

     • If the /etc/ilogtail/users directory does not exist, create it manually.

     • Changes to user identifiers take effect within one minute.

   • To collect logs for multiple Alibaba Cloud accounts, create an account ID file for each account. For example:

     touch /etc/ilogtail/users/{Alibaba Cloud account ID 1}
     touch /etc/ilogtail/users/{Alibaba Cloud account ID 2}

   Windows

   • In the C:\LogtailData\users directory, create a file using the Alibaba Cloud account ID as the filename.

     • Use Windows PowerShell.

       ni C:\LogtailData\users\{Alibaba Cloud account ID}

     • Use Command Prompt (cmd).

       type nul > C:\LogtailData\users\{Alibaba Cloud account ID}

   • To collect logs for multiple Alibaba Cloud accounts, create an account ID file for each account.

   Container environment

   For Logtail deployed in an Alibaba Cloud Kubernetes cluster (logtail-ds 1.7.3 or later), set the user identifier on the Container Service for Kubernetes console. Go to the Add-ons page and modify the LogtailDSExternalUserDefinelDs parameter of the logtail-ds component. Manage components.

   Note

   • To configure a user identifier, provide only the filename, without an extension.

   • You can configure multiple user identifiers on a server, but a Logtail container supports only one.

   • To remove a user identifier, delete the corresponding Alibaba Cloud account ID file from the server.

Step 2: Create a machine group

Collection configuration

A collection configuration defines rules for collecting, parsing, filtering, and processing data. It binds to a machine group to apply the rules to all servers in that group. Processing runs on the servers and consumes local resources. It has three parts:

• global configuration: Contains the Logtail configuration name, log topic, and tags for labeling and classifying collected logs.

  Global configuration parameters

  Parameter	Description
  Configuration Name	The name of the Logtail configuration. Must be unique within the project. Cannot be changed after creation.
  Log Topic Type	Specifies how to generate the log topic. Machine Group Topic: Simple Log Service allows you to apply a Logtail Configurations to multiple Machine Groups. You can use the Machine Group Topic to distinguish logs from different machine groups. When Logtail reports data, it uses the machine group's Machine Group Topic as the log topic and uploads it to the Project. You must specify the log topic as a query condition when querying logs. File Path Extraction: If different users or applications save logs in different top-level directories but with identical subdirectory and file name structures, Simple Log Service cannot distinguish which user or application generated the logs. In this case, use File Path Extraction to differentiate log data. This method uses a regular expression to match the full file path, and the matched result, such as a username or application name, is uploaded to Simple Log Service as the log topic. File path extraction scenarios Note In the regular expression for the file path, you must escape the forward slash (/). Scenario 1: Different users record logs in different directories, but the log files have the same name. The directory structure is as follows: /data/logs ├── userA │ └── serviceA │ └── service.log ├── userB │ └── serviceA │ └── service.log └── userC └── serviceA └── service.log If you only set the file path to /data/logs and the file name to service.log in the Logtail Configurations, Logtail collects the content from all three service.log files into the same LogStore. This makes it impossible to distinguish which user generated which log. You can use a regular expression to extract values from the file path to generate different log topics. Regular expression \/data\/logs\/(.*)\/serviceA\/.* Extraction result __topic__: userA __topic__: userB __topic__: userC Scenario 2: If a single log topic is insufficient to identify the log source, you can configure multiple regular expression capture groups in the log file path to extract key information. There are two types of capture groups: named capture groups (?P<name>) and unnamed capture groups. If you use a named capture group, the generated tag field is __tag__:{name}. If you use an unnamed capture group, the generated tag field is __tag__:__topic_{i}__, where {i} is the index of the capture group. Note When a regular expression contains multiple capturing groups, the __topic__ field is not generated. For example, if the file path is /data/logs/userA/serviceA/service.log, you can extract multiple values from the file path in the following ways. Example 1: Using unnamed capture groups in a regular expression. Regular expression \/data\/logs\/(.*?)\/(.*?)\/service.log Extraction result __tag__:__topic_1__: userA __tag__:__topic_2__: serviceA Example 2: Using named capture groups in a regular expression. Regular expression \/data\/logs\/(?P<user>.*?)\/(?P<service>.*?)\/service.log Extraction result __tag__:user: userA __tag__:service: serviceA Verification: After the configuration is complete, query the logs based on the log topic. On the Query and Analysis page, enter the generated log topic, such as __topic__: userA or __tag__:__topic_1__: userA, to query logs for the corresponding topic. Query and search syntax. On the Query and Analysis page of the Simple Log Service console, enter the query statement __topic__: userA and click Query / Analysis. The query result shows 14 log entries, which verifies that the topic extraction configuration has taken effect. The log details show that the path is /data/logs/userA/serviceA/service.log and the user field is userA. Custom: Enter customized://custom_topic_name to use a custom static log topic.
  Advanced Parameters	Additional advanced parameters are available in Create a Logtail pipeline configuration.

• input configuration: Defines the data source type (file input, container stdout, SQL query, or HTTP input) and specifies the collection path and source details for each type.

  Input configuration parameters

  Parameter	Description
  File Path	Specify the directory and name of log files based on the location of the logs on your server, such as an Elastic Compute Service (ECS) instance. Linux file paths must start with a forward slash (/). Example: /apsara/nuwa/**/app.Log. Windows file paths must start with a drive letter. Example: C:\Program Files\Intel\**\*.Log. You can specify an exact directory and an exact name. You can also use wildcard characters to specify the directory and name. When you configure this parameter, use only the asterisk (*) or question mark (?) as wildcard characters. Simple Log Service scans all levels of the specified directory to find the log files that match the specified conditions. Examples: If you specify /apsara/nuwa/**/*.log, Simple Log Service collects logs from the log files suffixed by .log in the /apsara/nuwa directory and its recursive subdirectories. If you specify /var/logs/app_*/**/*.log, Simple Log Service collects logs from the log files that meet the following conditions: The file name is suffixed by .log. The file is stored in a subdirectory of the /var/logs directory or one of its recursive subdirectories. The name of the subdirectory matches the app_* pattern. If you specify /var/log/nginx/**/access*, Simple Log Service collects logs from the log files whose names start with access in the /var/log/nginx directory and its recursive subdirectories.
  Maximum Directory Monitoring Depth	Specify the maximum number of levels of subdirectories that you want to monitor. The subdirectories are in the log file directory that you specify. This parameter specifies the levels of subdirectories that can be matched by the ** wildcard characters included in the value of File Path. A value of 0 specifies that only the log file directory that you specify is monitored.
  File Encoding	Select the encoding format of log files.
  First Collection Size	Specify the size of data that Logtail can collect from a log file the first time it does so. Default value: 1024. Unit: KB. If it's less than 1,024 KB, Logtail collects data from the beginning of the file. If it's equal to or greater than 1,024 KB, Logtail collects the last 1,024 KB of data in the file. You can configure First Collection Size based on your business requirements. Valid values: 0 to 10485760. Unit: KB.
  Collection Blacklist	If you enable this, configure a blacklist to specify the directories or files that you want Simple Log Service to skip when it collects logs. You can specify exact directories and file names. You can also use wildcard characters to specify directories and file names. When you configure this parameter, you can use only the asterisk (*) or question mark (?) as wildcard characters. Important If you use wildcard characters to specify a value for File Path and you want to skip some subdirectories in the specified directory, configure Collection Blacklist to specify the subdirectories. You must specify complete ones. For example, if you set File Path to /home/admin/app*/log/*.log and you want to skip all subdirectories in the /home/admin/app1* directory, select Directory Blacklist and enter /home/admin/app1*/** in the Directory Name field. If you enter /home/admin/app1*, the blacklist does not take effect. When a blacklist is in use, computational overhead is generated. We recommend a maximum of 10 entries per blacklist. You cannot specify a directory that ends with a forward slash (/). For example, if you specify the /home/admin/dir1/ directory, the directory blacklist does not take effect. The following types of blacklists are supported: File Path Blacklist If you select File Path Blacklist and enter /home/admin/private*.log in the File Path Name field, all files prefixed by private and suffixed by .log in the /home/admin/ directory are skipped. If you select File Path Blacklist and enter /home/admin/private*/*_inner.log in the File Path Name field, all files suffixed by _inner.log in the subdirectories prefixed by private in the /home/admin/ directory are skipped. For example, the /home/admin/private/app_inner.log file is skipped, but the /home/admin/private/app.log file is not. File Blacklist If you select File Blacklist and enter app_inner.log in the File Name field, all files whose names are app_inner.log are skipped. Directory Blacklist If you select Directory Blacklist and enter /home/admin/dir1 in the Directory Name field, all files in the /home/admin/dir1 directory are skipped. If you select Directory Blacklist and enter /home/admin/dir* in the Directory Name field, all files in the subdirectories prefixed by dir in the /home/admin/ directory are skipped. If you select Directory Blacklist and enter /home/admin/*/dir in the Directory Name field, all files in the dir subdirectory in each second-level subdirectory of the /home/admin/ directory are skipped. For example, the files in the /home/admin/a/dir directory are skipped, but those in the /home/admin/a/b/dir directory are not.
  Allow File to Be Collected Multiple Times	By default, you can use only one Logtail configuration to collect logs from a log file. If you want to collect multiple copies of logs from a log file, turn on Allow File to Be Collected Multiple Times.
  Advanced Parameters	Optional. Configure the advanced parameters that are related to input processors. For more information, see CreateLogtailPipelineConfig.

• Processing configuration: Uses processing plugins to parse and format collected data (filtering, desensitization, regex matching, JSON parsing).

  Processing configuration parameters

  Parameter	Description
  Log Sample	Add a sample log collected from an actual scenario. Use the sample log to easily configure parameters related to log processing. You can add multiple sample logs. Ensure that their total length does not exceed 1,500 characters. [2023-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened at TestPrintStackTrace.f(TestPrintStackTrace.java:3) at TestPrintStackTrace.g(TestPrintStackTrace.java:7) at TestPrintStackTrace.main(TestPrintStackTrace.java:16)
  Multi-line Mode	Specify the type of multi-line logs. A multi-line log spans multiple consecutive lines. You can configure this parameter to identify each multi-line log in a log file. Custom: A multi-line log is identified based on the value of Regex to Match First Line. Multi-line JSON: Each JSON object is expanded into multiple lines. Example: { "name": "John Doe", "age": 30, "address": { "city": "New York", "country": "USA" } } Configure Processing Method If Splitting Fails. Exception in thread "main" java.lang.NullPointerException at com.example.MyClass.methodA(MyClass.java:12) at com.example.MyClass.methodB(MyClass.java:34) at com.example.MyClass.main(MyClass.java:½0) For the preceding sample log, Simple Log Service can discard the log or retain each single line as a log if it fails to split it. Discard: The log is discarded. Retain Single Line: Each line of log text is retained as a log. A total of four logs are retained.
  Processing Method	Add processors as needed. You can add native and extended processors for data processing. Important Refer to the console page prompts for usage restrictions on the processors. Logtail V2.0 You can arbitrarily combine native processors for data processing. You can combine native and extended processors. Extended processors must follow native processors in the sequence. Logtail earlier than V2.0 You cannot add native and extended processors at the same time. You can use native processors only to collect text logs. When you add them, note the following: You must first add one of the following Logtail processors: Data Parsing (Regex Mode), Data Parsing (Delimiter Mode), Data Parsing (JSON Mode), Data Parsing (NGINX Mode), Data Parsing (Apache Mode), and Data Parsing (IIS Mode). After you add the first processor, you can add a Time Parsing processor, a Data Filtering processor, and multiple Data Masking processors. When you configure the Retain Original Field if Parsing Fails and Retain Original Field if Parsing Succeeds parameters, you can use only the following parameter combinations. For others, Simple Log Service does not ensure configuration effects. Upload logs that are parsed. Upload logs that are obtained after successful parsing, and raw ones if the parsing fails. Upload logs obtained after parsing. Add a raw log field to the logs if the parsing succeeds, and raw logs if it fails. For example, if a raw log is "content": "{"request_method":"GET", "request_time":"200"}" and it's successfully parsed, the system adds a raw log field, which is specified by the New Name of Original Field parameter. If you do not configure the parameter, the original field name is used. The field value is {"request_method":"GET", "request_time":"200"}.

Create a collection configuration

A collection configuration must bind to an active machine group before deployment. Create one based on the data source type: Log data collection.

Modify a collection configuration

Always modify a collection configuration at its source. Changes made through a different channel can be overwritten during maintenance operations like Pod recreation or component updates, causing incorrect log formats and collection interruptions.

Modification guidelines by creation method:

• Configurations created in the SLS console, CLI, or SDK: Modify directly with the same tool.

• Configurations created using the Container Service console or container environment variables: Modify the container environment variables, such as aliyun_logs_*. For parameter details, see Collect container logs from an ACK cluster. Modification in the SLS console is not supported.

  • Environment variables support only basic collection path settings, not advanced parameters or processing plugins. To use advanced features, delete the current environment variables and collection configuration, then recreate it using the SLS console, SDK, CLI, or a CRD.

  • If you later added processing plugins or advanced parameters in the SLS console, a Pod recreation or component update can overwrite those changes. Migrate your configuration promptly.

• Configurations created using a CRD: Directly edit the corresponding CR, such as AliyunPipelineConfig. For parameter details, see AliyunPipelineConfig parameters. Modification in the SLS console is not supported.

  • If you modified an AliyunPipelineConfig-based configuration in the SLS console, all changes revert within 30 minutes. Verify your configuration meets expectations.

  • If you modified an AliyunLogConfig-based configuration in the SLS console, a Pod recreation or component update can overwrite those changes. Update the CR promptly and consider upgrading to AliyunPipelineConfig.

Machine groups and collection configurations

SLS supports many-to-many bindings between collection configurations and Machine Groups. Configurations bind to machine groups, not directly to servers. When you add or remove servers, SLS automatically applies or removes the bound configurations. Servers running different operating systems cannot share a Machine Groups.