通过消费组读取文本日志，监控日志模板的统计信息-Simple Log Service(SLS)-阿里云帮助中心

Prerequisites

You have collected logs to the source Logstore or Metricstore. For more information, see Data collection and Ingest time-series data.

Important
The logstores that are described in this topic are Standard logstores. For more information, see Manage a logstore.
You have created an intelligent analysis instance.

Create a log template matching job

Log on to the Simple Log Service console.
Go to the Create Job page.
1. In the Log Application section, on the Intelligent O&M tab, click Intelligent Anomaly Analysis.
2. In the instance list, click the target instance.
3. In the left-side navigation pane, choose Analysis Tasks > Text Analysis > Log Template Matching.
4. In the upper-right corner of the Log Template Matching Task page, click Create Now.
In the Basic information section of the Create Template Matching Job wizard:
1. Configure parameters such as Task name, Project, Logstore type, and Source Logstore. The job writes all results to a Logstore named internal-ml-log in the current Project.
2. Create the AliyunLogETLRole role. Log Service assumes this role to obtain data. Click Next.

In the Algorithm configuration section of the Create Template Matching Job wizard, configure the algorithms. After you finish the configuration, click Finish. The Log Template Matching job starts to analyze logs from the configured start time.

Similarity clustering

This algorithm performs online clustering and matching analysis based on log similarity and does not depend on a log template repository.

Parameter	Description
Log entity fields	Groups log data based on the values of the specified log entity fields. The job performs template discovery separately for the log data in each group. You can select or enter a maximum of two fields.
Log level field	The field used to identify the risk level or severity of a log, such as `info` and `error`. If the log level field fails to parse or is empty, the system sets the log level to `unknown`.
Log text fields	Select the log fields to analyze. The Text Analysis job concatenates the values of the selected fields and treats them as a single unit for statistical analysis. You can select a maximum of five log fields. If a target field is not in the drop-down list, you can enter it manually. Important You do not need to create an index. Log Template Discovery and Log Template Matching analyze text content. The values of the corresponding log fields must be of the `text` type. Otherwise, the system automatically converts the values to the `text` type. If none of the specified fields have a value, the job does not perform statistical analysis on the corresponding logs.
Time window length	The algorithm analyzes logs in each time window by using a sliding window. Unit: seconds. The algorithm waits for all logs in a time window to be collected before it performs analysis and generates results.
Anomaly count threshold	If the number of anomalous events in a time window reaches this threshold, the entity is considered anomalous. You can configure this threshold in one of the following ways: Set as a relative value. The value must be in the range of `[0.1,0.5]`. Set as an absolute value. The value must be in the range of `[1,20]`.
Similarity threshold	If the similarity between a log and a log category template is greater than the similarity threshold, the log is classified into that log category. A lower similarity threshold increases the likelihood of grouping logs into the same category.

Advanced parameters

Parameter	Description
Number of initialization windows	The number of windows required for the cold start of a Text Analysis job, which includes log accumulation and algorithm preparation. The log similarity clustering algorithm only accumulates logs and prepares the algorithm in the initial windows. It performs anomaly detection on the accumulated logs in subsequent time windows. For more information, see How do I set the number of initialization time windows?.
Maximum mute windows	Use Maximum mute windows and Time window length to specify the maximum mute duration for detecting rare log categories. For a discovered log category, if no logs from that category are collected within the maximum mute duration, the Text Analysis job considers the category lost. If logs of the same category appear again later, the job defines it as a new log category. For example, a system outputs Error logs during a failure. After the system recovers, these logs no longer appear. If the system fails again after the maximum mute duration has passed, it outputs Error logs again, and the job reports a new log category. The default value is 672.
Sampling rate	The proportion of logs that the Text Analysis job processes. The default value is 1, which means all logs are processed. If the sampling rate is less than 1, the Text Analysis job randomly selects the specified proportion of logs for processing. This parameter is suitable for scenarios with massive log volumes. If the log volume exceeds the processing capacity of the job, you can decrease the value of this parameter.
Delimiter	The Text Analysis job tokenizes logs by using the specified delimiters and whitespace characters. For example, if a log is `11:22:33:44:55` and the delimiter is a colon (:), the log content is parsed into 11, 22, 33, 44, and 55.
Data filtering configuration	Filters logs based on the log level field. If the risk level of a log matches a log level in the whitelist, the Text Analysis job analyzes the log. If the risk level of a log matches a log level in the blacklist, the Text Analysis job does not analyze the log.
General field template	During log preprocessing, the similarity clustering algorithm uses a template expression to match text content in logs and replaces it with a template name. This improves the accuracy of the analysis. For example, if you set Template Name to `IP` and Template Expression to `((?<=[^A-Za-z0-9])\|^)(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})((?=[^A-Za-z0-9])\|$)`, text that matches the expression is replaced with `IP`. The template expression must follow Python regular expression rules. You can configure a maximum of five templates. Enter the text to test and click Test to verify that the configured field template works as expected. For example, if you enter 192.0.2.0, the match result is `<:IP:>`, which indicates that the field template is configured correctly.

Similarity matching

This algorithm depends on a log template repository. It matches log data against templates in the repository and stores the match result statistics.

Parameter	Description
Template source Project	The Project that contains the log template repository.
Template source Logstore	The Logstore that contains the log template repository.
Template source task	The resource ID of the task that contains the log template repository.
Similarity	If the similarity between a log and a log category template is greater than the similarity threshold, the log is classified into that log category. A lower similarity threshold increases the likelihood of grouping logs into the same category.
Log entity fields	Groups log data based on the values of the specified log entity fields. Template matching is performed separately for the log data in each group. You can select or enter a maximum of two fields.
Log level field	The field used to identify the risk level or severity of a log, such as `info` and `error`. If the log level field fails to parse or is empty, the system sets the log level to `unknown`.
Log text fields	Select the log fields to analyze. The Text Analysis job concatenates the values of the selected fields and treats them as a single unit for statistical analysis. Configure a field whitelist: The job processes only the fields selected in the whitelist. Configure a field blacklist: The job ignores the fields selected in the blacklist and processes only the other fields in the logs. Important You do not need to create an index. Log Template Discovery and Log Template Matching analyze text content. The values of the corresponding log fields must be of the `text` type. Otherwise, the system automatically converts the values to the `text` type. If none of the specified fields have a value, the job does not perform statistical analysis on the corresponding logs.

Advanced parameters

Parameter	Description
Matching method	Select a matching method. String matching: Matches logs and log templates by comparing characters directly. This method is suitable for scenarios with a small number of logs and templates. Vector matching: Vectorizes logs and templates and then matches them. This method is suitable for scenarios with a large number of logs and templates. Hash matching: Matches logs and log templates by using similarity hashing. This method provides fuzzy matching and is suitable for scenarios with a very large number of logs and templates.
Maximum tokens	The maximum number of tokens to process per log. The job ignores content that exceeds this limit.
Initial constant length	Specifies that the first N tokens in each log line must be part of the final log template. For example, if you set this parameter to 2, the algorithm assumes that the first two tokens in each log line are part of the final log template.
Sampling rate	The proportion of logs that the Text Analysis job processes. The default value is 1, which means all logs are processed. If the sampling rate is less than 1, the Text Analysis job randomly selects the specified proportion of logs for processing. This parameter is suitable for scenarios with massive log volumes. If the log volume exceeds the processing capacity of the job, you can decrease the value of this parameter.
Delimiter	The Text Analysis job tokenizes logs by using the specified delimiters and whitespace characters. For example, if a log is `11:22:33:44:55` and the delimiter is a colon (:), the log content is parsed into 11, 22, 33, 44, and 55.
Data filtering configuration	Filters logs based on the log level field. If the risk level of a log matches a log level in the whitelist, the Text Analysis job analyzes the log. If the risk level of a log matches a log level in the blacklist, the Text Analysis job does not analyze the log.
General field template	During log preprocessing, the algorithm uses a template expression to match text content in logs and replaces it with a template name. This improves the accuracy of the analysis. For example, if you set Template Name to IP and Template Expression to `((?<=[^A-Za-z0-9])\|^)(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})((?=[^A-Za-z0-9])\|$)`, text that matches the expression is replaced with `IP`. The template expression must follow Python regular expression rules. You can configure a maximum of five templates. Enter the text to test and click Test to verify that the configured field template works as expected. For example, if you enter 192.0.2.0, the match result is `<:IP:>`, which indicates that the field template is configured correctly.

Hash clustering

This algorithm does not depend on a log template repository. It performs online clustering and matching analysis based on the log clustering feature.

Parameter	Description
Similarity threshold	If the similarity between a log and a log category template is greater than the similarity threshold, the log is classified into that log category. A lower similarity threshold increases the likelihood of grouping logs into the same category.
Merging threshold	If the similarity between two log category templates is greater than the merging threshold, the job attempts to merge the two templates into one. A lower merging threshold increases the likelihood of merging templates.
Time window length	The algorithm analyzes logs in each time window by using a sliding window. Unit: seconds. The algorithm waits for all logs in a time window to be collected before it performs analysis and generates results.
Maximum number of categories	The maximum number of log templates that the job can save. If this limit is exceeded, the job deletes automatically generated templates.
Data latency	The duration the job waits before processing log data in each time window to ensure that the log clustering results are generated.
Retry wait time	The duration that the job waits before it retries reading the log clustering results if an error occurs.
Delimiter	The Text Analysis job tokenizes logs by using the specified delimiters and whitespace characters. For example, if a log is `11:22:33:44:55` and the delimiter is a colon (:), the log content is parsed into 11, 22, 33, 44, and 55.

Parameter	Description

	LogReduce
	LogReduce

View log matching job results

Log on to the Simple Log Service console.
Go to the Job Details page.
1. In the Log Application section, on the AIOps tab, click Intelligent Anomaly Analysis.
2. In the instance list, click the target instance.
3. In the left-side navigation pane, choose Analysis Task > Text Analysis > Log Template Matching.
4. Click the target job ID to view the matching results.
Matching results vary by algorithm type.
Similarity clustering algorithm

Note
After you create a job, there is a delay before the analysis results for the first time window are available. The waiting time depends on the log volume within the time window and typically does not exceed the duration of a time window.
1. On the Log Template Management tab, view the generated log templates. You can also click Overview to view log count trends.
2. On the Log Anomaly Details tab, view the details of log anomalies. If you configured the Log Entity Field when you created the text analysis job, you can filter by entity in the upper-left corner of the page. The analysis results for the selected entity include the following:
  - Log count trend statistics: Shows the changes in log counts for each time window in the selected time range, and displays an anomaly score when the count changes abnormally.
  - Log category anomaly overview: Shows overall and per-category log anomalies for each time window in the selected time range. A higher score indicates a more severe anomaly. Scores range from 0 to 1.
  - Log anomaly details: Click a specific colored block to display the details of the corresponding anomaly event in the Log Anomaly Details area. This area displays all anomaly events within the selected time range, including the log category, log level, anomaly type, anomaly score, and the anomaly's time window. For more information about anomaly types, see Anomaly Type Description.
Similarity matching and hash clustering

View the matching status of each log template on the matching job dashboard.

At the top of the dashboard are four metric cards: Matched Log Count, Unmatched Log Count, Matched Log Count for Selected Group, and Unmatched Log Count for Selected Group. In the lower-left corner is the Log Matching Distribution in Selected Group pie chart, and on the right is the Log Templates in Selected Group table. The table includes the pattern_id, pattern, labels, and status columns. You can use the Log Group selector at the top to switch between groups.

The Log Matching Details page is divided into three sections. In the upper-left corner, the Change in Log Matching for Selected Group table shows the pattern_id, the current number of matches (current), the number of matches from the previous period (previous), and the period-over-period change rate (ratio) for each log template. Abnormal change rates are highlighted in red or yellow. On the right, the Trend of Log Template Matching line chart plots the match count (cnt) over time. At the bottom, the Online Clustering Results for Selected Group table lists the clustered log templates (pattern) along with corresponding examples (example).

Configure alerts for log template matching

Note

You can configure log alerts only for the similarity clustering algorithm.

Log on to the Simple Log Service console.
Go to the Job Details page.
1. In the Log Application section, on the Intelligent O&M tab, click Intelligent Anomaly Analysis.
2. In the instance list, click the target instance.
3. In the left-side navigation pane, choose analysis task > Text Analysis > Log Template Matching.

Configure alerts for log templates.

In the upper-right corner of the page, click Alert Configuration.
In the Alert Configuration dialog box, configure alerts for existing templates or preset alerts for new templates, and then click OK.
1. When you set Select Template Configuration to Configure Alerts for Existing Templates, you can specify a log template by template ID and log level and then set an alert.
2. When you set Select Template Configuration to Preset Alerts for New Templates, you can specify log templates by log level and then set an alert. An alert is triggered when a new template with the specified log level appears.

The following table describes the alert parameters.

Parameter	Description
Alert name	A custom name for the alert.
Template ID	The ID of the log template for which to configure an alert. If you specify both Template ID and Log Level, the conditions are combined with an OR operator.
Log level	The log level of the templates for which to configure an alert.
Alert condition	The condition that triggers the alert. Count Condition: An alert is triggered when the number of logs that match the log template exceeds the specified threshold. Abnormal Fluctuation in Count: Triggers an alert when the count of matching logs fluctuates abnormally. Rare Log Occurrence: Triggers an alert when a rare log template is detected.
Trigger condition	If you set Alert Condition to Count Condition, you must configure the log count threshold.
Alert policy	An alert policy merges, mutes, and suppresses generated alerts. When you select Simple Mode or Standard Mode, you do not need to configure an alert policy. By default, Log Service uses the built-in dynamic alert policy (sls.builtin.dynamic) to manage alerts. When you select Advanced Mode, you can select a custom action policy and alert policy. For information about how to create an alert policy, see Create an alert policy.
Action policy	An action policy controls settings such as the notification channels and frequency for alerts. When you set Alert Policy to Simple Mode, you only need to configure an action group. After you configure the action group, Log Service automatically creates an action policy named `Rule Name-Action Policy`. This action policy sends all alerts triggered by this rule. For information about how to configure notification channels, see Notification channels. Important You can modify this action policy on the Action Policy management page. For more information, see Action policies. If you add conditions when you modify the action policy, the Alert Policy in this context automatically changes to Standard Mode. When you set Alert Policy to Standard Mode or Advanced Mode, you can select a built-in or custom action policy for alert notifications. For information about how to create an action policy, see Action policies. If you set Alert Policy to Advanced Mode, you can also enable or disable Custom Action Policy. For more information, see Dynamic action policy mechanism.