Problem description
CDN and DCDN services may experience abnormal traffic growth. This issue often results from HTTP requests that have missing or empty Referer fields.
These empty Referer requests typically originate from malicious scraping. Attackers use scripts or tools to directly access CDN resource URLs, bypassing normal page reference relationships. This can cause the following problems:
-
Unexpected increases in CDN traffic costs, resulting in additional expenses.
-
Excessive consumption of origin server bandwidth, affecting normal business operations.
-
Unauthorized use of content resources, harming business interests.
This issue commonly occurs in the following use cases:
-
URLs of static resources, such as images, videos, or files, are leaked or scraped by crawlers.
-
Attackers use automated tools to batch download resources.
-
Third-party websites or applications directly reference resources without authorization.
Problem analysis
Definition of empty Referer
An empty Referer request refers to one of two states of the Referer field in the HTTP request header:
-
Missing: The HTTP request header does not contain the Referer field.
-
Empty: The HTTP request header contains the Referer field, but its value is an empty string (
Referer:).
How do I know if it is empty Referer scraping?
View the CDN console to see whether recent traffic spikes are caused by malicious requests with empty Referer.
Go to CDN console > Analytics > Operations Reports > Popular Referer Headers. Review the traffic proportion of empty Referer requests. If this proportion is too high, such as over 50%, and inconsistent with the business baseline, scraping may be occurring.
In the report, records where Refer is - indicate empty Referer requests.
In the time filter, select the time range during which the traffic was abnormal and click Query to determine if empty referer requests account for the highest traffic share.
Solutions
Use ESA to prevent hotlinking. ESA provides CDN acceleration capabilities and supports WAF configuration, enabling more comprehensive mitigation. The following section describes two configuration methods based on whether your service allows empty Referer requests:
-
Use case 1: Business does not allow any empty Referer requests. Configure a WAF custom rule in ESA to block all empty Referer requests. Configuration path: Site > Security Protection > WAF > Custom Rules.
Expression:
(exists(http.referer) and len(http.referer) lt 1) or (not exists(http.referer))When a request matches the rule, set the action to Block, the response code to 403, and select the default block page.
-
Use case 2: Business needs to allow some empty Referer requests, such as homepage access. Configure WAF rules to block empty Referer requests, and add whitelist rules to allow specific paths, such as the home page. Proceed as follows:
-
Configure a custom rule to block empty Referer. Configuration path: Site > Security Protection > WAF > Custom Rules.
Use the same expression as in Scenario 1:
(exists(http.referer) and len(http.referer) lt 1) or (not exists(http.referer)). When a request matches the rule, set the action to Block, the response code to 403, and select the default block page. -
Configure a whitelist rule to allow specific paths, such as the home page. Configuration path: Site > Security Protection > WAF > Whitelist Rules.
Expression:
(http.host in {"your hostname"} and http.request.uri in {"/"}). On the Add Whitelist Rule page, set the match condition where Hostname equals your target domain (for example,your hostname), and add a second condition with an And operator where URI equals/. In the Then skip section, select All rules, and then click OK.
-