Analyze offline logs-CDN(CDN)-阿里云帮助中心

You can use a command-line tool to parse the offline log file and extract information such as the top 10 IP addresses by request volume, user agents, and referrers. This topic shows how to analyze CDN offline access logs with a command-line tool in a Linux environment.

Collect and prepare log data

See Quick Start to obtain the log file aliyundoc.com_2018_10_30_000000_010000.gz.

Upload the log file to a local Linux server.

Sample log entry:

[9/Jun/2015:01:58:09 +0800] 10.10.10.10 - 1542 "-" "GET http://www.aliyun.com/index.html" 200 191 2830 MISS "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://example.com/robot/)" "text/html"

Log on to the local Linux server and run the following command to decompress the log file.
```
gzip -d aliyundoc.com_2018_10_30_000000_010000.gz
```
Decompression creates a log file named aliyundoc.com_2018_10_30_000000_010000.
Naming rule for log files: Accelerated domain name_year_month_day_start time_end time[extension field].gz. The extension field starts with an underscore (_). Example: aliyundoc.com_2018_10_30_000000_010000_xx.gz.
Note
Names of specific log files may not contain an extension field. Example: aliyundoc.com_2018_10_30_000000_010000.gz.

Identify and filter abnormal behavior

Check request volume

You can analyze the number of IP requests in the offline access log data to identify abnormal request volume. Abnormal request volume typically has the following characteristics:

Abnormally high request volume: Conduct an in-depth review of the access frequency from a single source IP address. If an IP address is detected to have initiated a request volume that significantly deviates from the baseline within a specific time period, this is highly suspected to be traffic abuse.
Large volume of requests in a short period: A sudden traffic spike or an unusual, cyclical request pattern.
List the top 10 IP addresses by request volume.
```
cat [$Log_Txt] | awk '{print $3}' |sort|uniq -c|sort -nr |head -10
```
Note
- awk '{print $3}': Extracts the third column of the log file, which is the IP address. Columns are separated by spaces.
- sort: Sorts the IP addresses.
- uniq -c: Counts the occurrences of each IP address.
- sort -nr: Sorts the results by count in descending order.
- head -n 10: Retrieves the top 10 IP addresses with the most requests.
- [$Log_Txt]: Replace with the name of your log file, for example, aliyundoc.com_xxxxxxx.

User agent analysis

Analyze the user agent in the offline access log to identify abnormal requests. Abnormal user agents typically have the following characteristics:

Abnormal or forged user agent: Many scraping tools use default or forged user agents. You can find these by filtering for uncommon, suspicious, or empty user agents.
Extract and count user agents.
```
 grep -o '"Mozilla[^"]*' [$Log_Txt] | cut -d'"' -f2 | sort | uniq -c | sort -nr | head -n 10
```
You can filter out common user agents to identify suspicious ones.
```
grep -v -E "Firefox|Chrome|Safari|Edge" [$Log_Txt]
```
Count the number of requests with an empty user agent.
```
awk '!/Mozilla/' [$Log_Txt] | wc -l
```
Note
grep -o: Outputs only the matched content.
grep -v -E: Inverts the match to show lines that do not match the specified patterns.
wc -l : Counts the number of lines.

Request pattern analysis

Analyze the requested URLs in the offline access log to identify abnormal request patterns. Abnormal URL requests typically have the following characteristics:

High URL similarity: Traffic scraping often involves a large number of requests to similar or identical URLs. Analyzing URL patterns can help you detect these abnormal requests.
High request ratio for specific resource types: An unusually high number of requests for specific resources, such as images, CSS, and JS files, may indicate traffic scraping.
List the top 10 URLs by request volume.
```
grep -oP 'https?://[^" ]+' [$Log_Txt] | sort | uniq -c | sort -nr | head -n 10
```

Status code analysis

Analyze the status codes in the offline log to identify abnormal requests.

Look for high ratio of 4xx or 5xx responses: A high number of 4xx or 5xx status codes from a single IP address may indicate malicious crawling attempts.

Count the occurrences of different status codes.

awk '{print $9}' [$Log_Txt] | sort | uniq -c | sort -nr

List the top 10 IP addresses that generated 400 status codes.

grep ' 400 ' [$Log_Txt] | awk '{print $3}' | sort | uniq -c | sort -nr | head -n 10