Log search

Query and analyze logs

1. Log on to the Web Application Firewall (WAF) console. In the top menu bar, select the resource group and region for your WAF instance: Chinese Mainland or Outside Chinese Mainland.

2. In the left navigation pane, choose Security Operations > Log Service.

3. From the domain name drop-down list, select the domain name, and then turn on the Status switch.

   The domain name drop-down list (① in the figure) contains only the domain names that are protected by WAF. If you have not added a domain name to WAF, add one first. For more information, see Tutorials.

4. On the Log Search tab, use search and analytic statements to query and analyze WAF log data.

   1. In the search box (① in the figure), enter a search statement.

      Search statements use the syntax specific to Log Service. For more information about the syntax, see Search syntax and functions. You can use WAF log fields as query fields in your search statements. For a list of supported fields, see WAF log fields.

      If you are not familiar with the search syntax, use Advanced Search. Expand Advanced Search above the search box, set the search conditions, and then click Search. A search statement matching your conditions is automatically generated. The following table describes the available search conditions.

      Search condition	Description
      IP	The client's IP address.
      Request ID	The unique ID that WAF generates for a client request. WAF provides this ID when it returns a block page or a slider CAPTCHA challenge. You can use this ID to troubleshoot issues.
      Rule ID	The ID of the WAF protection rule triggered by the request. You can find rule IDs on the Security Report page or on the Systems > Protection Rule Group page.
      Status Code Returned from Origin Server	The HTTP status code returned by the origin server.
      Status Code Returned from WAF	The HTTP status code that WAF returns to a client.
      Protection Features	The type of WAF protection rule that is matched by the request. For more information about WAF protection modules and how to configure rules for each module, see Overview.

   2. To perform statistical analysis on the query results, append an analytic statement after the search statement in the search box (① in the figure). If you only want to query logs that meet specific conditions, skip this step.

      Separate the analytic statement from the search statement with a vertical bar (|). Analytic statements use standard SQL-92 syntax. For more information about analytic statements, see Overview of log query and analysis.

   3. Use the time picker (② in the figure) to specify a time range for the log query.

   4. Click Search & Analyze (③ in the figure).

      The query and analysis results, which are the WAF logs that match your query, are displayed at the bottom of the page. The results include a log distribution histogram, Raw Log, and a Graph. You can perform quick analysis, generate graphs, and create alerts based on the results. For more information, see Manage query and analysis results and Create an alert.

   For more examples of log queries and analysis, see Query and analysis examples.

Manage query and analysis results

Create an alert

You can create an alert based on the current search and analytic statement. After you create an alert, Log Service periodically evaluates the query results. If the results meet the specified trigger condition, Log Service sends you a notification. This lets you monitor your service status in real time.

On the query and analysis page, choose Save as Alert > New Alert to create an alert based on the query results. For more information, see Configure alerts.

Query and analysis examples

• Analyze the number of attack requests blocked by different WAF protection modules at 15-minute intervals. The results show the time (time) and the number of requests blocked by the following modules: the WAF rules engine (wafmodule), the IP address blacklist and custom Access Control List (ACL) policies (aclmodule), and HTTP flood protection (httpfloodmodule).

  * |
  SELECT
    time_series(__time__, '15m', '%H:%i', '0') as time,
    COUNT_if(final_plugin = 'waf') as "wafmodule",
    COUNT_if(final_plugin = 'acl') as "aclmodule",
    COUNT_if(final_plugin = 'cc') as "httpfloodmodule"
  GROUP by
    time
  ORDER by
    time

• Analyze the distribution of protection module types (final_plugin) that triggered the final action. The results show the number of hits (times), the requested domain name (host), and the final protection module (final_plugin).

  * |
  SELECT
    count(*) as times,
    host,
    final_plugin
  GROUP by
    host,
    final_plugin
  ORDER by
    times desc

• Analyze the queries per second (QPS) at 15-minute intervals. The results show the time (time) and QPS.

  * |
  SELECT
    time_series(__time__, '15m', '%H:%i', '0') as time,
    count(*) / 900 as QPS
  GROUP by
    time
  ORDER by
    time

• Find the domain names most frequently targeted by HTTP flood attacks. The results show the number of blocked requests (times) and the requested domain name (host).

  *
  and acl_action :block |
  SELECT
    count(*) as times,
    host
  GROUP by
    host
  ORDER by
    times desc

• Analyze website request log details in 1-second intervals. The results show the time (time), the requested domain name (host), the request path (request_path), the request method (request_method), the WAF response status code (status), the origin server response status code (upstream_status), and the query string (querystring).

  * |
  SELECT
    date_format(date_trunc('second', __time__), '%H:%i:%s') as time,
    host,
    request_path,
    request_method,
    status,
    upstream_status,
    querystring
  LIMIT
    10

• Query the 10 most recent attack request records for a website (your_domain_name). The results show the request time (time), the real client IP (real_client_ip), and the client type (http_user_agent).

  matched_host: your_domain_name
  and final_action: block |
  SELECT
    time,
    real_client_ip,
    http_user_agent
  ORDER by
    time desc
  LIMIT
    10

• Analyze the number of days (days_passed, rounded to one decimal place) that have elapsed since an attack request to a website (your_domain_name) was blocked by WAF.

  matched_host: your_domain_name
  and final_action: block |
  SELECT
    time,
    round((to_unixtime(now())-__time__) / 86400, 1) as "days_passed",
    real_client_ip,
    http_user_agent
  ORDER by
    time desc
  LIMIT
    10

• Analyze the daily trend of attack requests to a website (your_domain_name).

  matched_host: your_domain_name
  and final_action: block |
  SELECT
    date_trunc('day', __time__) as dt,
    count(1) as PV
  GROUP by
    dt
  ORDER by
    dt

  The date_trunc function is used to group timestamps by day. For more information about this function, see Date and time functions.

• Analyze the distribution of source countries (country) for attack requests to a website (your_domain_name).

  matched_host: your_domain_name
  and final_action: block |
  SELECT
    ip_to_country(
      if(real_client_ip = '-', remote_addr, real_client_ip)
    ) as country,
    count(1) as "Number of attacks"
  GROUP by
    country

  In WAF logs, the real_client_ip field represents the real client IP. If the real client IP cannot be obtained for reasons such as user access through a proxy server or an incorrect IP field in the request header (in which case the value of real_client_ip is -), you can use the remote_addr field (which represents the remote address) as the real client IP.

• Analyze the distribution of source provinces (province) for attack requests to a website (your_domain_name).

  matched_host: your_domain_name
  and final_action: block |
  SELECT
    ip_to_province(
      if(real_client_ip = '-', remote_addr, real_client_ip)
    ) as province,
    count(1) as "Number of attacks"
  GROUP by
    province

  The ip_to_province function retrieves the province information for a real client IP. For more information about this function, see IP functions.