New log clustering
New log clustering in Simple Log Service groups similar logs into structured log templates and analyzes the variables within those templates. Use it to identify anomalies, locate root causes, and track log changes across time periods. The feature is designed for troubleshooting, version regression testing, and security auditing.
Overview
Key concepts
New log clustering is built on the following concepts:
Log category: A group of logs that share a similar format. Logs in different formats belong to different log categories.
Log template: The common pattern extracted from logs in the same log category. A template consists of constants and variables.
Log variable: The dynamic part of a log template. Variables are usually the runtime information recorded by a log output statement and are highlighted in the template.
Log constant: All content of a log template except the variables. Constants are identical for all logs in the same log category.
Core capabilities
New log clustering provides the following capabilities:
Log pattern extraction: Automatically discovers log categories in large volumes of logs and extracts a log template for each category.
Variable distribution analysis: Displays the distribution of variable values within a log template.
Grouped clustering: Groups logs by specified fields before clustering, which is useful for analyzing logs from multiple modules.
Comparative analysis: Compares log patterns between two time periods to identify anomalies. The current time period is the experiment group; the historical comparison period is the control group.
Reverse lookup with regular expressions: Provides the regular expression of a log template so that you can query the raw logs.
New log clustering computes log templates in real time on top of the field index. No additional index traffic is generated beyond the traffic of log field indexing.
Comparison with old log clustering
The following table compares new and old log clustering to help you choose the right version for your scenario.
| Feature | New log clustering | Old log clustering |
| Index traffic | No additional index traffic. Only the field index is required. | Generates additional index traffic |
| Processing efficiency | Real-time computation. Samples logs when the data volume is large. | High processing efficiency, which suits large data volumes |
| Algorithm accuracy | Uses an algorithm with higher accuracy | Relatively lower algorithm accuracy |
| Variable distribution | Supports analysis of variable distribution in log patterns | Not supported |
| Raw log lookup | Reverse lookup with the regexp_like operator | Natively supports direct lookup |
| Multi-field clustering | Supports only single-field clustering | Supports clustering on multiple fields at the same time |
| Grouped clustering | Supported | Not supported |
| Dashboard integration | Not supported yet | Supported |
| Clustering precision adjustment | Not supported | Supported |
Selection guidance
New log clustering: Suitable for scenarios where the log volume is moderate or the logs are already filtered, and fine-grained analysis such as variable distribution or comparative analysis is required. If you need to cluster logs on multiple fields simultaneously, use old log clustering instead.
Old log clustering: Suitable for scenarios where large volumes of logs must be clustered quickly and the results must be displayed on a dashboard. Also required when you need multi-field clustering or clustering precision adjustment.
Prerequisites
Before you begin, make sure that the following requirements are met:
- New log clustering clusters text fields for which indexing is enabled.
Run a log clustering analysis
This section walks you through a complete log clustering analysis, including basic clustering, grouped clustering, and comparative analysis.
Step 1: Go to the LogReduce page
Log on to the Simple Log Service console.
In the Projects list, click the target project.
In the left-side navigation pane, choose
Log Storage, and then click 
in front of the target LogStore name to expand the drop-down menu.In the drop-down menu, click
Query and Analysis.On the query and analysis page, click LogReduce > Real-time Clustering - New.
Step 2: Configure clustering parameters and run the analysis
On the clustering page, configure the following parameters:
Set the analysis scope
Time range (optional): Select the time period of the logs that you want to analyze.
Query statement (optional): Enter a query statement to filter logs. Narrowing the analysis scope to exclude irrelevant logs, such as
* and not level:INFO, improves the accuracy and efficiency of the clustering analysis.
Select the clustering target
Clustering field (required): Select the text field to cluster. Simple Log Service recommends the field that best suits clustering based on the field content. You can select only one field.
(Optional) Grouped clustering and comparative analysis
Aggregation field (optional): Select one or more fields to pre-group the logs before clustering. For details about how to choose aggregation fields, see the About aggregation fields section below. You can select a maximum of three aggregation fields.
Comparison time (optional): Select a time period for comparison. Comparing with historical data helps you find log patterns that are new, have disappeared, or have changed in volume during a version release or a service failure.
Common scenarios:
Comparing logs before and after a version release.
Comparing a failure period with a normal period.
Periodic comparison, such as today against the same period yesterday, or this week against the same period last week.
Evaluating the results of a performance optimization.
Configuration methods:
Time offset: Shift the comparison period forward or backward relative to the current time range.
Custom time: Manually select the start time and end time of the control group.
Configuration recommendations:
Select time periods of the same duration and avoid overlapping time ranges.
For periodic services, use a year-on-year or period-on-period comparison, such as the same period yesterday or last week.
Combine the comparison with aggregation fields for a layered analysis. For example, compare log changes for each service by using the service dimension.
About aggregation fields
Aggregation fields are the dimension fields used for pre-grouping. Logs are aggregated by these fields before clustering.
When to use aggregation fields:
Logs from different modules differ significantly and must be clustered separately.
You want to view clustering results grouped by a dimension such as log level or service name.
Recommended fields: Log level, service, module, and environment. Select fields that have a limited number of distinct values and are meaningful to your business.
Tips:
You can select multiple aggregation fields (up to three) for combined grouping.
Pre-aggregation improves clustering accuracy and prevents logs from different dimensions from being mixed into the same category.
Limit: You can select a maximum of three aggregation fields.
Step 3: Interpret the clustering results
After the analysis is complete, the results appear in a list. Each row represents a log template.
Clustering Mode: The pattern extracted from the logs. Variables in the pattern are highlighted.
Quantity: The total number of logs that match the template.
Log Distribution: A histogram that shows how the logs of the template are distributed over time.
Group information: If you configured an aggregation field, this column displays the group that the template belongs to.
The clustering algorithm samples a representative batch of logs for analysis. If the time window contains a large volume of logs with complex formats, the sampled logs may not cover all log categories. You can configure a query statement to filter out irrelevant logs and improve the clustering results.
Step 4: Drill down into the variable distribution
After you find a suspicious log template, for example, an error log template whose count surges, analyzing the distribution of its variables helps you locate the root cause.
In the list of clustering results, click a highlighted variable in the Clustering Mode column.
In the panel that appears, view the distribution of the variable:
Enumerated values: The top N values of the variable and the number of occurrences of each value. For example, you can view the distribution of the
error_codevariable to quickly locate the most frequent error code.Numeric values: The distribution of the numeric range of the variable.
New log patterns: Present in the experiment group but not in the control group.
Disappeared log patterns: Present in the control group but not in the experiment group.
Significant volume changes: Log patterns whose count differs notably between the two periods.
Step 5: Look up the raw logs
After you locate a specific log template, you can view the complete raw logs for more context.
In the list of clustering results, click the row of the target log template to go to the template details page.
On the Log Sample tab, view the raw logs that match the template. Up to 50 logs are displayed.
To query all logs that match the template:
In the upper-right corner, click Display Parsing Rules to obtain the regular expression of the template.
Copy the regular expression.
Go to the Query and Analysis page and run a query with the
regexp_likeoperator. ReplaceContentwith the name of your clustering field:
* | SELECT * FROM log WHERE regexp_like(Content, 'your_regular_expression')Best practices
Focus on abnormal logs
Use a query statement to exclude normal logs before clustering. This reduces noise and improves clustering accuracy. After the analysis runs, focus on log patterns with abnormal volumes or patterns that are new.
Compare across releases
Set the current time range to the period after a release and the comparison time to the period before the release. Focus on patterns that are new or have disappeared, and check the variable distribution to determine whether anomalies exist.
Isolate modules with aggregation fields
Set the aggregation field to a module identifier such as Component or ServiceName. This clusters each module separately so that you can quickly locate the module with the problem.
FAQ
Why are the clustering results empty?
Possible causes:
The selected field contains no data. Verify that the clustering field contains data, or adjust the time range.
The query statement filtered out all logs. Verify that the query statement is correct.
Indexing is not enabled for the field. Make sure that an index is configured for the clustering field.
What can I do if the variables in a log template are not identified accurately?
New log clustering uses algorithms to identify variables automatically. Inaccurate identification may occur when:
The log format is irregular, and the constants are not clearly distinguished from the variables.
The log sample size is insufficient. Expand the time range or reduce the filter conditions.
How do I view all logs of a log category?
Follow the steps in Step 5: Look up the raw logs to obtain the regular expression of the template and query all matching logs on the Query and Analysis page.
Why do some log patterns appear in only one time period during a comparative analysis?
This usually indicates one of the following situations:
Unique to the experiment group (current period): The logs may be new. Check whether they indicate an anomaly.
Unique to the control group (comparison period): The logs may have disappeared because an issue was fixed or a feature was changed.
Analyze the results together with your business context to determine whether further investigation is required.
What's next
Use the
regexp_likeoperator to build automated log analysis pipelines based on the log templates identified through clustering.If you need dashboard integration for your clustering results, use old log clustering, which supports display on dashboards. For more information, see Log clustering.