Query and analyze application logs-Simple Log Service(SLS)-阿里云帮助中心

This topic describes how to use Simple Log Service to query and analyze application logs in different scenarios, such as log query, association analysis, and statistical analysis.

Background information

Application logs include important statistical information about application operations and maintenance. Application logs have the following characteristics:

Inconsistent log styles. Application developers have different styles when they write code. Therefore, logs that are generated by different applications have inconsistent styles.
Large data size. The size of application logs is one order of magnitude larger than the size of access logs.
Distribution on multiple servers. Most applications are stateless and run on different frameworks, such as cloud servers and container services. These applications may be deployed on a few to thousands of instances. Therefore, a cross-server solution for log collection is required.
Complex runtime environments. Applications are running in different environments, and application logs are stored in different environments. For example, application-related logs are stored in containers, API-related logs are stored in Function Compute, old system logs are stored in data centers, mobile app logs are stored in mobile terminals, and website logs are stored in browsers.

Before you can obtain full logs, make sure that all application logs are stored in the same environment. Simple Log Service provides multiple log collection methods and the log analysis feature. You can analyze logs from end to end in real time by using the query statements and SQL-92 syntax. You can also visualize the query results in charts. The query and analysis solution that is provided by Simple Log Service in this topic costs only 25% of an open source solution.

Query application logs

For example, an app encounters an order error or request latency. You can use a search statement to query logs that contain terabytes of data and locate the issue within 1 second. You can also specify a query time range and keywords in the search statement to obtain more precise results.

You can execute the following search statement to query the logs of POST requests whose latency exceeds 1 second:
```
Latency > 1000000 and Method=Post*
```
You can execute the following search statement to query the logs that include the error keyword and exclude the merge keyword:
```
error not merge
```

Perform association analysis on application logs

Association analysis supports intra-process association analysis and cross-process association analysis. The two types have the following differences:

Intra-process association analysis: Intra-process association is typically straightforward because logs for a single process are typically in one file. In a multi-threaded environment, you can simply filter by thread ID.
Cross-process association analysis: By default, the cross-process association is unclear if multiple processes are used to handle a request. In this case, you can initiate a remote procedure call (RPC), which carries the TracerId parameter. You can use this parameter to associate the processes.

Intra-process association analysis
Use a contextual query to view related logs. For example, you can find an error log by searching with a keyword, and then click Context View to see the N preceding and N following log entries. For more information about the procedure, see Contextual query. On the Raw Logs tab in Simple Log Service, select the Raw display mode, and then click the Context View icon (magnifier icon) to the left of the target log entry to view its context and perform intra-process association analysis.
Contextual query results: On the Context View page, the current log entry (at position 0) is highlighted. The context range includes 3 log entries before and 3 log entries after the current entry (-3 to +3). The page also provides Highlight and Filter input fields to help you quickly locate key information.
Cross-process association analysis
Cross-process association analysis is also referred to as tracing. Common tracing tools include EagleEye, Dapper, Stackdriver Trace, Zipkin, Appdash, and X-Ray.
You can use Simple Log Service to perform basic tracing. To do this, output a correlation identifier, such as request_id or OrderId, in the logs of each module. You can then search for this identifier across different Logstores to find all related logs. For example, in the search box on the Simple Log Service (SLS) query page, enter the request_id value 595F9A92D30E8E444FC971F4 and set the time range to 15 Minutes (Relative). The query returns 15 associated logs with a status of Exact Results. If you expand the details of the first log, you can see that the Nginx access log for this request has a status code of status:404, a log time of 22/Oct/2020:11:08:52, a source IP of 172.16.64.165, and a log file path of /root/testnginx/acce...1.log. By using a request_id, you can perform cross-process log association analysis to quickly locate the complete path of a problematic request.
For example, you can use an SDK to query logs from frontend servers, backend servers, payment systems, and order systems. After you retrieve the results, you can create a frontend page to visualize the cross-process trace. In a tracing system, when you select a specific trace ID, you can view the complete call stack as a tree. This view shows the call hierarchy and execution time for each method. For example, KVEngine::QueryInterval, which is called by ServerBase::HandleRequest, takes 61935 μs and accounts for 98.99% of the total time, revealing it as the main performance bottleneck. The method YouchaoFile::CreateBlock takes 19603 μs, or 31.32% of the total time. By analyzing the trace, you can quickly pinpoint performance bottlenecks in cross-process requests.

Perform statistical analysis on application logs

After you query and obtain logs, you can perform statistical analysis on the logs.

You can execute the following query statement to analyze the types of all errors and the distribution of all errors by location:

__level__:error | select __file__, __line__, count(*) as c group by __file__, __line__ order by c desc

错误类型和位置

What to do next

Back up logs
You can back up the obtained logs to other cloud services, such as Object Storage Service (OSS) and MaxCompute.
Configure alerts
- Configure alerts in Simple Log Service
- Configure alerts in CloudMonitor
Grant permissions
You can grant permissions to a RAM user to isolate the development environment and the production environment.