Log FAQ

• How can I reduce storage usage for Log Analysis?

• Can I export Cloud Firewall traffic logs to a third-party system?

• How do I check my remaining log storage capacity?

• Why do traffic logs contain periodic ICMP probe traffic from Alibaba Cloud?

• Why do some traffic logs show the application as "Unknown"?

• Is Log Analysis data retained after I release Cloud Firewall?

• Can I export log audit records?

• How can I use logs to find the total number of attacks blocked by Cloud Firewall?

• Why are some persistent connection traffic logs missing after a Cloud Firewall engine upgrade?

• Why is the Cloud Firewall log count different from that of Anti-DDoS Proxy or WAF?

How to reduce Log Analysis storage

You can reduce log storage usage by shortening the log storage duration, delivering fewer log types, shipping logs to an OSS bucket, or clearing stored logs.

• Reduce the log storage duration

  After you enable Log Analysis, the default log storage duration is 180 days. If you do not require such a long retention period, you can shorten the duration. For more information, see Modify the log storage duration.

• Reduce the number of delivered log types

  Cloud Firewall enables log delivery for all log types by default. To reduce storage, deliver only the log types you need. For more information, see Configure collected log types.

• Periodically ship logs to an OSS bucket

  If you generate large volumes of logs and must retain them all, we recommend shipping them to an OSS bucket. For more information, see Create an OSS data shipping task (new version).

• Clear logs

  If you have large volumes of test logs that are no longer needed, we recommend clearing them. For more information, see Manage log storage capacity.

Can I export logs to a third-party system?

Yes. You can use the Log Analysis feature to export logs and then ingest the exported log files into a third-party system, such as a security operations center (SOC).

Choose an export method based on the volume of your logs.

• Small log volumes

  You can use the download feature in Log Analysis to save logs locally and then upload them to the third-party system. For more information about how to download logs, see Export logs.

• Large log volumes

  You can use the Simple Log Service (SLS) console and consumer groups to programmatically export log data to a third-party system. For more information, see Use a consumer group to consume logs.

How to check remaining log storage

You can check your log storage capacity only after enabling Log Analysis. Once enabled, you can view your used and remaining capacity in the Cloud Firewall console. For more information, see Manage log storage capacity.

At the top of the Log Analysis page in the Cloud Firewall console, a progress bar displays your used capacity, total capacity, and usage percentage. You can also click Upgrade Capacity or Clear to manage your capacity.

Why are there periodic ICMP probe logs?

Cloud Firewall periodically sends Internet Control Message Protocol (ICMP) packets for health checks to ensure service quality. These health checks are not attacks and will not affect your services.

You can log on to the Cloud Firewall console and find the source IP addresses of these SLA probes in the cloud service address book named Source address for SLA monitoring. For more information, see Address book.

Why do logs show "Unknown" applications?

When Cloud Firewall cannot identify the application of a traffic flow, it labels the application as "Unknown". This can happen for several reasons:

• The session consists of fewer than three packets and is not fully established. This is common for scan traffic.

• A Layer 4 access control policy blocked the traffic before a session could be established.

• The intrusion prevention system or another mechanism reset the connection, which prevented application signature matching.

• The traffic is encrypted, or it uses a non-standard protocol, an internal application, or a protocol not supported by deep packet inspection (DPI).

By default, Cloud Firewall allows unidentified traffic to avoid service disruptions. If you want to block this traffic, you can enable strict mode for the relevant firewall. For more information, see Access control engine modes or Configure the access control engine mode for NAT firewalls.

Is log data retained after release?

Data retention depends on the instance billing method. Data is not retained for Subscription instances but is retained for Pay-as-you-go instances.

After you release a Cloud Firewall instance, its configuration data, such as access control policies, attack protection policies, and traffic analysis policy configurations, is retained for 7 days. If you have a Subscription instance and need to retain your Log Analysis data, you must export the logs to a local machine or a third-party system before you release the instance. For more information, see Export logs.

Can I export log audit records?

You cannot export log audit records directly. However, you can search for and export the underlying raw logs from the Log Analysis page by using a query statement.

For example, to export inbound internet firewall traffic logs from the last 24 hours for the HTTPS application, follow these steps:

1. Log on to the Cloud Firewall console.

2. In the left-side navigation pane, choose Detection & Response > Log Analysis.

3. On the Log Analysis tab, enter the query statement in the search box and set the time range to 1 Day. For more information, see Query and analyze logs.

   The following is the query statement:

   log_type:internet_log and direction:"in" and app_name:"HTTPS"

4. Export the query results. For more information, see Export logs.

How to count blocked attacks in logs

You can set the query statement to rule_result:drop and specify a time range. The Log Quantity in the query results indicates the total number of attacks blocked by Cloud Firewall. For more information, see Query and analyze logs.

Why do log counts differ across services?

This discrepancy is expected because these services log traffic at different network layers. Cloud Firewall logs traffic at Layer 4 (the transport layer), recording inbound and outbound traffic. Anti-DDoS Proxy and WAF log traffic at Layer 7 (the application layer), recording HTTP requests.

Layer 4 logs focus on individual TCP or UDP connections and packets, while Layer 7 logs focus on complete HTTP requests and responses. A single HTTP request may be split into multiple TCP packets, resulting in multiple Layer 4 log entries but only one Layer 7 log entry.

Furthermore, Layer 4 logging is more verbose due to factors such as packet retransmissions, network latency, fragmentation, and application-layer protocols such as HTTP Keep-Alive. When you compare or analyze logs from these different layers, it is important to consider their inherent differences.

Missing logs for persistent connections after an upgrade

Cloud Firewall engine maintenance, such as upgrades or scaling, is designed to be non-disruptive to your services. However, during these events, log reporting for some existing persistent connections may be interrupted, resulting in missing log entries.

To identify potentially affected traffic, you can query for logs of existing connections by using the filter new_conn=0 on the Log Analysis page with a 1-minute time range. Logging for these connections may be incomplete. When the connections are re-established, logging resumes as normal.

For more information about how to enable Log Analysis, see Enable Log Analysis.

How to query service-linked role logs

1. Log on to the ActionTrail console.

2. In the left-side navigation pane, choose Events > Event Query.

3. In the top navigation bar, select the region where the event occurred.

4. From the filter drop-down list, select Operator, set its value to aliyunserviceroleforcloudfw, specify a time range, and then click the search icon to query for related events.

5. Find the desired event and click View Details in the Actions column to view its details and code. For more information, see Query events.