The Intelligent Anomaly Analysis application of Simple Log Service analyzes text content in logs and provides global statistical analysis results. You can create log pattern discovery and log pattern matching jobs to monitor and analyze logs, and select a job and algorithm based on your log characteristics.
Overview of text analysis algorithms
In a log pattern discovery job, you can set up a log pattern library offline by using the log clustering algorithm or pattern discovery algorithm. In a log pattern matching job, you can monitor logs online by using the similarity clustering algorithm, hash clustering algorithm, or similarity matching algorithm.
The text analysis algorithms use LogParser and anomaly detection technologies. Log analysis reports help you understand global log information and potential anomalies.
-
The reports help you identify log categories that may have anomalies, narrowing down the scope for manual troubleshooting. These categories include new log categories and the top five log categories with the highest anomaly scores.
-
You can review the reports regularly to track changes in global log information and monitor system stability.
Log pattern discovery
The log clustering algorithm is suitable for large volumes of logs with consistent formats. The pattern discovery algorithm is suitable for moderate volumes of logs with complex formats.
Log clustering algorithm
The log clustering algorithm builds on the log clustering feature, which performs coarse-grained clustering on logs. The algorithm then performs fine-grained clustering based on those results. For more information about how to enable the log clustering feature and view clustering results, see LogReduce.
Pattern discovery algorithm
The pattern discovery algorithm uses word frequency analysis to cluster logs with similar high-frequency words into one category. The high-frequency words form the log pattern of that category. For more information about the algorithms, see Efficient and Robust Syslog Parsing for Network Devices in Datacenter Networks.
Log pattern matching
The similarity clustering algorithm and hash clustering algorithm are suitable for large volumes of logs with consistent formats. The similarity matching algorithm is suitable for large volumes of logs.
Similarity clustering algorithm
The similarity clustering algorithm uses text similarity-based LogParser to parse text logs and classify similar logs into one category based on content and structure. Text similarity metrics include edit distance, Jaccard similarity, and cosine similarity. The algorithm further analyzes log changes across continuous time windows by category to detect potential anomalies. For more information about the algorithm, see Drain: An Online Log Parsing Approach with Fixed Depth Tree.
Hash clustering algorithm
The hash clustering algorithm builds on the log clustering feature, which clusters logs online. The algorithm then performs further clustering based on those results and continuously analyzes and monitors logs. For more information, see LogReduce. The hash clustering algorithm does not use external log pattern libraries.
Similarity matching algorithm
The similarity matching algorithm uses external log pattern libraries to match and analyze logs. It can also use log pattern libraries built in a log pattern discovery job. This algorithm collects statistics on the occurrences of each log pattern and identifies new log patterns at the earliest opportunity. It accelerates log pattern matching by using methods such as vector matching and hash matching.