Defend against abnormal UA attacks with custom rules

Network architecture overview

The following figure shows the network architecture for this example. WAF blocks malicious traffic and ensures that legitimate user traffic reaches your ECS instances.

Quick deployment

To test how WAF defends against abnormal User-Agent attacks, you can use the quick deployment feature to quickly set up the required network architecture.

Abnormal User-Agent attacks

An abnormal User-Agent attack occurs when an attacker forges or alters the User-Agent header in an HTTP request to hide their identity or bypass security measures. Common types of these attacks include:

• Malicious crawlers: Scrape large amounts of website content, consuming bandwidth and resources.

• Vulnerability scanning: Automatically scan your website for exploitable security flaws.

• HTTP flood attacks: Send a high volume of requests to overload the server, making it unable to respond to legitimate traffic.

• Spoofing and evasion: Forge the User-Agent to bypass specific security policies or access controls.

Procedure

Step 1: Collect and analyze User-Agent data

Before you create protection rules, you need to understand the User-Agent strings used by legitimate visitors. This helps you distinguish between normal and malicious traffic. In this example, access logging for the Layer-7 Classic Load Balancer instance is enabled, allowing you to view and analyze the logs in Simple Log Service. To learn how to enable and analyze CLB logs, see Enable data collection.

1. Log on to the Simple Log Service console. Select your destination project and Logstore to open the log query page.

2. After you enable log collection for the Classic Load Balancer, Simple Log Service collects request information that passes through the instance. For details about log fields, see Log fields for Layer-7 access logs of a CLB instance. The http_user_agent field contains the User-Agent string from the request. Click the http_user_agent field and select the most frequent User-Agent value. The console then displays all logs from the specified time range that contain this value.

   

   Requests with the User-Agent Apache-HttpClient/4.5.13 (Java/1.8.0_381) account for 99% of the traffic during this period. Within 15 minutes, 4,368 requests have this User-Agent string: Apache-HttpClient/4.5.13 (Java/1.8.0_381).

Step 2: Create a WAF custom rule

If you find a domain name or API receiving a high volume of requests with forged or abnormal User-Agent strings, such as null values or random strings, you need to identify these requests based on your business scenario.

In this example, the data from Step 1 shows a sudden surge in requests from a single User-Agent, far exceeding normal levels. This indicates potential malicious traffic. Follow these steps to create a custom rule that blocks requests matching these conditions:

1. Log on to the Web Application Firewall 3.0 console. From the top menu bar, select the resource group and region (Chinese Mainland or Outside Chinese Mainland) for the WAF instance.

2. In the navigation pane on the left, choose Protection Config > Core Web Protection.

3. On the Core Web Protection page, navigate to the Custom Rule section and click Create Template.

4. On the Create Template page, click Create Rule to add a custom rule to the template.

   1. Rule Name: Enter Abnormal_UA_Interception.

   2. Match Condition: Set Match Field to User-Agent, Contains to Contains, and for Match Content, enter Apache-HttpClient/4.5.13 (Java/1.8.0_381).

      Enter the content that best fits your specific business requirements to ensure the rule is both effective and precise.

   3. Action: Select Block.

   

5. Click OK. The custom rule is created. Make a note of the rule ID, which you can use later to check its effectiveness in security reports. In this example, the rule ID is 20766535.

   

6. On the Create Template page, in the Available Objects section, select the Protected Objects or Protected Object Groups that you want to protect and add them to the Selected Objects section.

   

7. Click Added.. When the Added successfully message appears, the custom rule template is created.

Verification

After the custom rule takes effect, you can verify its effectiveness in several ways. You can inspect request information in the logs and view the rule's performance in security reports.

View WAF logs

In WAF logs, you can find the abnormal User-Agent requests that WAF blocked. A final_action value of block indicates that WAF blocked the request. For more information about log fields, see Log fields.

1. Log on to the Web Application Firewall 3.0 console. In the top navigation bar, select the resource group and region (Chinese Mainland or Outside Chinese Mainland) where your WAF instance resides.

2. In the left-side navigation pane, choose Detection and Response > Log Service.

3. At the top of the Log Service page, select the protected object that corresponds to your Classic Load Balancer. The following figure shows that WAF blocks traffic with the abnormal User-Agent and records it in the logs.

   

View WAF security reports

You can view the effectiveness of your custom rule in the WAF security reports.

• Log on to the Web Application Firewall 3.0 console.

• In the left-side navigation pane, choose Detection and Response > Security Reports. On the Security Reports page, use the filters at the top to select the Custom Rule module and the protected object for your Classic Load Balancer.

  In the Top 5 Hits section, you can see how many times the rule you created in Step 2 has triggered.

View application logs

After the rule takes effect, the logs on your application server will no longer show requests with the abnormal User-Agent string. In this example, the Classic Load Balancer access logs shown in the figure confirm that the instance no longer receives requests with the abnormal User-Agent within a 15-minute period.