Read text logs from consumer groups for template discovery, extraction, and management-Simple Log Service(SLS)-阿里云帮助中心

Prerequisites

Logs are collected to a source Logstore or Metricstore. For more information, see Data collection and Collect time series data.

Important
The logstores that are described in this topic are Standard logstores. For more information, see Manage a logstore.
An intelligent analysis instance is created.

Create a log template discovery job

Log on to the Simple Log Service console.
Navigate to the job creation page.
1. In the Log Application area, on the Intelligent Operations and Maintenance (O&M) tab, click Intelligent Anomaly Analysis.
2. In the instance list, click the target instance.
3. In the navigation pane on the left, choose Analysis Task > Text Analytics > Log Template Discovery.
4. In the upper-right corner of the Log Template Discovery Task area, click Create Now.
In the Basic Information step of the Create Template Discovery Job wizard, perform the following operations:
1. Configure parameters such as Task Name, Project, Logstore Type, and Source Logstore. All task results are written to a Logstore named internal-ml-log in the current Project.
2. Create the AliyunLogETLRole role. Simple Log Service assumes this role to obtain data. Click Next.

In the Algorithm Configuration step of the Create Template Discovery Job wizard, configure the required algorithms.

Log clustering algorithm

Parameter	Description
Number of templates	Move the slider to control the number of log templates to be generated. You cannot precisely control the number of log templates, but you can adjust the approximate range.
Query statement	The filter condition for logs that are used for LogReduce. Template discovery is performed on logs that meet the filter condition.

Template discovery algorithm

Parameter	Description
Log entity field	Groups log data based on the values of the log entity field. Template discovery is performed separately on the log data in each group. You can select or enter up to two fields.
Log level field	The field used to identify the risk level or severity of a log, such as `info` or `error`. If an error occurs when the log level field is parsed or the field is empty, the log level is `unknown`.
Log text field	Select the log fields to be analyzed. The text analytics job concatenates the values of the selected fields and analyzes them as a whole. If the target field is not in the drop-down list, you can enter it manually. If you configure a blacklist, the algorithm analyzes all log fields except for the blacklisted fields. If you configure a whitelist, the algorithm analyzes only the whitelisted fields. Important An index is not required. Log template discovery and log template matching analyze text content. The value of the corresponding log field must be of the `text` type. Otherwise, the value is automatically converted to the `text` type. If the values of all specified fields do not exist, the text analytics job does not perform statistical analysis on the corresponding log.
Number of templates	Move the slider to control the number of log templates to be generated. You cannot precisely control the number of log templates, but you can adjust the approximate range.
Start time, End time	The time range of the logs to be analyzed.

Advanced Configuration

Parameter	Description
Maximum number of templates	The maximum number of templates that the log template discovery algorithm can discover.
Maximum number of entities	The maximum number of log entities (log groups) to process. If the number of entities that appear in the logs exceeds this value, new log entities are ignored. The more entities there are, the longer the template discovery algorithm takes.
Maximum number of tokens	Limits the maximum number of tokens for each log after tokenization. The part of a log that exceeds this limit is ignored.
Starting constant length	Specifies that a certain number of words at the beginning of a log must be part of the final log template. For example, if you set this parameter to 2, the algorithm considers the first two words of each log to be part of the final log template.
Sample rate	Adjusts the proportion of logs processed by the text analytics job. The default value is 1, which means all logs are processed. If the sample rate is less than 1, the text analytics job randomly selects the corresponding proportion of logs for processing. This parameter is suitable for scenarios with massive amounts of logs. If the log volume exceeds the processing capacity of the job, you can reduce this parameter.
Separator	The text analytics job uses separators, including the separators configured here and whitespace characters, to tokenize logs. For example, if a log is `11:22:33:44:55` and the separator is a colon (:), the log content is parsed into 11, 22, 33, 44, and 55.
Data filtering configuration	Filters logs by log level field. If the risk level of a log matches a log level in the whitelist, the text analytics job analyzes the log. If the risk level of a log matches a log level in the blacklist, the text analytics job does not analyze the log.
General field template	When a text analytics job preprocesses logs, the log similarity clustering algorithm uses a template expression to match text content in the logs and replaces the content with the template name. This helps improve analysis accuracy. For example, if Template Name is IP and Template Expression is `((?<=[^A-Za-z0-9])\|^)(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})((?=[^A-Za-z0-9])\|$)`, the text content that matches the template expression is replaced with `IP`. Template expressions must follow Python regular expression rules. You can configure up to five templates. Enter the text content to test and click Test to verify whether your field template configuration meets expectations. For example, if you enter 192.0.2.0, the matched result is `<:IP:>`, which indicates that the field template is configured as expected.

Click Complete. The created log template discovery job appears on the Job Resources tab.
Optional: In the Actions column for the target job resource, click the icon to edit or delete the job.

Important
A log template discovery job cannot be recovered after it is deleted. Perform this operation with caution.

View the progress of a log template discovery job

Log on to the Simple Log Service console.
Open the Job Details page.
1. In the Log Application area, on the Intelligent O&M tab, click Intelligent Anomaly Analysis.
2. In the instance list, click the target instance.
3. In the navigation pane on the left, choose Analysis Task > Text Analytics > Log Template Discovery.
4. On the Job Resources tab, click the target job ID to view the job's running progress, the number of discovered templates, and any exceptions that occurred during runtime.

View the results of a log template discovery job

Log on to the Simple Log Service console.
Open the Resource Details page.
1. In the Log Application area, on the Intelligent O&M tab, click Intelligent Anomaly Analysis.
2. In the instance list, click the target instance.
3. In the navigation pane on the left, choose Analysis Task > Text Analytics > Log Template Discovery.
4. On the Template Resources tab, find the required template resource based on its description and click its ID.
5. On the Template Resource page, you can view the log templates discovered by the job, and create, delete, modify, and annotate them.