Working principle of text analysis in Simple Log Service intelligent anomaly analysis-Simple Log Service(SLS)-阿里云帮助中心

Simple Log Service provides the text analysis feature to automatically detect anomalies in text content across large volumes of logs.

Important

Starting July 15, 2025 (UTC+8), the intelligent anomaly analysis feature will no longer be available to new users. Existing users can continue to use it.

Scope of impact

The following core features will be unpublished: intelligent health check, text analytics, and time series forecasting.
Feature Migration Solutions

The machine learning syntax, scheduled query and analysis (scheduled SQL), and dashboard features of SLS can fully replace the unpublished features.

Background information

Running services generate large volumes of system and business logs for monitoring and troubleshooting. Traditional log analysis relies on assessing risk levels and matching keywords such as Error, Failed, and Unsuccessfully. In distributed microservices environments, this approach presents the following challenges:

Terabytes or even petabytes of logs are generated daily, making manual analysis labor-intensive.
In a distributed environment with microservices deployment, Warning logs or Error logs do not necessarily indicate system exceptions. These logs may be generated due to system scaling, updates, or iterations. Professional knowledge is required during manual analysis to identify anomalies in logs.

Automated and intelligent log analysis addresses these challenges by reducing labor costs and unlocking the full potential of log data. This type of analysis has the following characteristics:

Processes large volumes of logs efficiently.
Identifies anomalies in logs or narrows down the scope of logs used for troubleshooting.
Allows you to customize parameters for text analysis.

Simple Log Service text analysis streamlines log integration and anomaly detection. You only need to configure the monitored objects and a few algorithm parameters. The algorithm then automatically detects anomalies, enabling you to focus on the important content.

Feature introduction

Text analysis pulls text content from logs through consumer groups without requiring indexes. Jobs retrieve data and feed it into the text analysis model on a configured schedule. The model writes results to the target logstore (internal-ml-log) and visualizes them on a dashboard.

Configure monitoring objects: Specify the log fields to analyze (the field values must be text content), configure algorithm parameters as needed, and start the task. Consumer group-based field configuration does not require indexes.
Scheduled data analysis: The algorithms in text analysis process data using time windows.
Result output: Analysis results are written to the target logstore, and a dashboard is generated to visualize them.

Terms

Term	Description
Job	A text analysis job includes data features and algorithm model parameters.
Instance	A text analysis job creates an instance based on the job configuration. The instance pulls data at regular intervals, runs the algorithm model, and distributes the results. Only one instance can run in a job at a time regardless of whether the instance is run on schedule or is retried due to an anomaly. You cannot concurrently run multiple instances in a single job. You cannot modify a job configuration while the job is running. If you modify the configuration, the job re-creates an instance and runs the new algorithm model. The new instance is independent of the previous one. For information about how different operations affect the scheduling and running of instances, see Scheduling and running.
Instance ID	Each instance is identified by a unique ID.
Creation time	The time when an instance is created. Typically, instances are created based on the scheduling rules of the job. If historical data needs to be processed or a previous instance timed out, an instance is created immediately.
Running time	The time when an instance starts running. If the job is retried, this value reflects the most recent start time.
End time	The time when an instance stops running. If the job is retried, this value reflects the most recent end time.
Status	The current state of an instance. Valid values: RUNNING STARTING SUCCEEDED FAILED
Algorithm configuration	The algorithm configuration includes the time window length, the threshold for the total number of anomalies, and the template source.
Text analysis result	The text analysis result includes the log template, log level, log category, configuration information, and anomaly score. For more information, see View the log template to check the job progress.

Scheduling and execution use cases

Each job can create one or more instances, but only one instance runs at a time. The following are common scheduling and execution use cases:

Immediate start: Starting a text analysis job immediately means the algorithm model can't access historical data. It trains on the data from the configured initialization time windows, suppressing anomalies, and then dynamically updates with new data.
Modified schedule parameters: When you change a job's scheduling rules, a new instance is generated based on the updated settings. The model continues from the last analyzed point, handling new data.
Retry on failure: If an instance fails due to issues like permission errors, unavailable log stores, or configuration problems, Simple Log Service auto-retries. If an instance remains in the STARTING state, it indicates a configuration failure, and an error log is created. Check the configuration and retry. The instance status updates to SUCCEEDED or FAILED based on the retry outcome.

Usage notes

To improve the efficiency of text analysis:

Specify only the text fields that you want to analyze. Including redundant fields may reduce analysis effectiveness and speed.
Review the time series data of monitored objects to check data stability and periodicity. This helps you configure appropriate algorithm parameters.