Prevent unauthorized CDN traffic with referer blocking
Background
This topic describes how to analyze access data to identify patterns of unauthorized traffic. It provides configuration methods, use cases, and recommendations for various protection solutions to help you effectively control sudden cost spikes caused by unauthorized resource usage.
Identify unauthorized traffic patterns
Basic query: offline logs
Download offline logs to view access logs for the relevant time period. Analyze the HTTP request details to identify suspicious IP addresses, User-Agents, and more. Offline logs contain limited data fields. If you want to view more data, use the real-time logs feature.
Use command-line tools to quickly parse the offline log files and extract information such as the top 10 IP addresses or User-Agents by request volume. For details, see Analysis methods for Alibaba Cloud CDN access logs.
Advanced query: operation reports and real-time logs
You must customize operation reports to enable statistical analysis. If you have already configured real-time log delivery or subscribed to operation reports, you can view historical log data. Operation reports are a free, built-in feature of Alibaba Cloud CDN.
To generate real-time logs, you must first enable Log Service (SLS) and ensure logs are delivered successfully. This is a paid feature. For pricing details, see Pricing Details.
You must configure both real-time logs and operation reports in advance. If you did not configure these features before the high charges occurred, you can use only offline logs for historical analysis.
After customizing operation reports, you can view user access reports, including PV/UV, Regions and ISPs, Domain Name Ranking, Popular Referer Headers, Popular URLs, Popular Origin URLs, and Top Client IPs. For details, see Customize and Subscribe to Operation Reports.
To query more log information, such as Referer and URI details, enable Log Service (SLS) to collect real-time logs. You are billed based on the number of delivered log entries after this feature is enabled.
Configure real-time log delivery for the accelerated domain name you want to analyze, as described in Configure real-time log delivery.
On the Real-time Logs page, find the project you want to analyze and click Log analysis.
On the Log Analysis page, filter the time range in the upper-right corner. Click the Raw Logs tab on the left, find the
refer_domainfield, and view the Referer information sorted in descending order.
Blocking strategies
Handle empty referers
If you experience a large volume of traffic from requests with an empty referer, you can use Edge Security Accelerator (ESA) to manage it.
If your business allows access from an empty referer, you can configure a JavaScript challenge for empty referer requests to block unauthorized traffic. If your business does not allow empty referer access, you can block these requests directly.
JavaScript challenge
-
In the ESA console, navigate to Websites. In the Website column, click the target site.
-
In the left-side navigation pane, choose .
-
Click the Custom Rules tab. On the Custom Rules tab, click Create Rule.
-
Enter a Rule Name.
-
In the If requests match... section, define the match conditions for requests. For more information, see Components of a rule expression.
-
In the Then execute... section, specify the action to perform on matching requests. For details, see Action descriptions.
-
-
Click OK.
-
For the match condition, click Edit Expression and enter the following expression:
(exists(http.referer) and len(http.referer) lt 1 and http.host eq "xxxxxx.aliyun.com") or (not exists(http.referer) and http.host eq "xxxxxx.aliyun.com")Replacexxxxxx.aliyun.comwith your actual domain name. -
Set the action to JavaScript challenge.
Block empty referers
Scenario 1: Your business does not allow any empty referer requests. Configure a WAF custom rule in ESA to block all empty referer requests. Path: Site > Security Protection > WAF > Custom Rules.
Expression:
(exists(http.referer) and len(http.referer) lt 1) or (not exists(http.referer))
Scenario 2: Your business needs to allow some empty referer requests (for example, homepage access). Configure a WAF rule to block empty referer requests, and then add an allowlist rule to allow access to specific paths, such as your homepage. Follow these steps:
Configure a custom rule to block empty referers. Path: Site > Security Protection > WAF > Custom Rules.
Configure an allowlist rule to allow a specific path (for example, the homepage). Path: Site > Security Protection > WAF > Allowlist Rule.
Expression:
(http.host in {"your hostname"} and http.request.uri in {"/"})
H5, miniapp, APP, and client scenario
This scenario extends the APP and client program scenario by also excluding miniapp requests. First, exclude miniapp and APP requests. Then, set allowed Referers for the remaining traffic.
Configure the rules engine to separate APP, client program, and miniapp UAs from other UAs. This example uses the Alipay mini program and
java-1.1.
Configure a Referer hotlink protection whitelist. Allow access only from specified Referers.
H5, APP, and client program scenario
Unlike browsers, APPs and client programs do not follow standard HTTP Referer behavior. They construct requests directly, so most requests lack a Referer. If you block all empty-Referer requests, you might disrupt normal APP traffic. Apply Referer-based blocking only to web traffic, not to APP or client requests.
For example, use a distinctive user agent (UA) string that differs from typical browser UAs. The following example uses the client program UA java-1.1.
Configure the rules engine to separate APP and client program UAs from other UAs.
Configure a Referer hotlink protection whitelist. Allow access only from specified Referers for web traffic.
Miniapp, APP, and client scenario
Miniapp UAs are relatively fixed but differ across vendors. Common examples include the following:
WeChat mini program: The key identifier in the UA is
MicroMessenger/x.x.x.Alipay mini program: The key identifier in the UA is
AlipayClient/x.x.x+ Nebula.Baidu mini program: The key identifiers in the UA are
BaiduApp/x.x.xorswan.ByteDance (Douyin): The key identifier in the UA is
BytedanceWebview/ Douyin.
First, identify the primary miniapp type your business uses, and then configure the settings accordingly. The rules engine determines the appropriate solution.
Configure the rules engine to separate APP, client program, and miniapp UAs from other UAs. This example uses the Alipay mini program and
java-1.1.Configure a Referer hotlink protection whitelist. Allow access only from specified Referers. Because miniapps and APPs rarely send real Referers, create a fake Referer that does not exist and add it to the whitelist. Apply this rule globally.
APP and client scenario
In a pure APP or client program environment, high-frequency traffic from the application's own UA may indicate an issue within the application. If the UA in the traffic does not match the application's UA, apply the same solution as described in the H5, APP, and client program scenario.
Configure the rules engine to separate APP and client program UAs from other UAs.
Configure a Referer hotlink protection whitelist. Allow access only from specified Referers. Because real Referers are absent, specify a placeholder Referer value that no real client sends and add it to the whitelist. Apply this rule globally.
Large file transfer scenario
Large file transfers are slow, infrequent, and lack predictable access patterns. This makes them difficult to protect. Even low-frequency access can consume large amounts of bandwidth.
Solution 1: Per-request rate limiting
Use per-request rate limiting to restrict requests from non-APP or non-miniapp clients.
Configure the rules engine to detect client features. Use the UA string for detection. This example uses the Alipay mini program and
java-1.1.Set the per-request rate limit and associate it with the rule that you created.
Solution 2: Authentication-based protection
If you can modify your server to add authentication, you can protect large files without limiting download speed. Choose between URL authentication and remote authentication.
URL authentication
This method requires modifying the large file URL on your server. Use the authentication logic of Alibaba Cloud CDN to generate time-limited URLs. For more information, see type A signing. After you update your server, configure the matching authentication settings in the CDN console.
Remote authentication
This method does not require changes to the request URL. Instead, deploy a dedicated authentication server that checks client features to decide whether to allow access.
Deploy a remote authentication server. Design the authentication logic to limit requests per client. For example, count requests per client IP over a 24-hour period and block access if the count exceeds your threshold.
In the CDN console, enter the authentication server address. Pass the client IP and any other required client information as parameters to the server.
Block malicious referers
By analyzing unauthorized traffic patterns, you can identify illegitimate referers. If you find malicious referers, use a referer blocklist to block them.
-
Log on to the CDN console.
-
In the left navigation pane, click Domain Names.
-
On the Domain Names page, find the target domain name and click Manage in the Actions column.
-
In the domain's navigation pane, click Access Control.
-
On the Referer Black/Whitelist tab, click Modify.
-
Set Type to Blacklist. In the Rules field, enter the list of malicious referers. Select Ignore Scheme.
-
Decide whether to select Allow Empty Referer based on your business needs.
-
Select (Allow): If your users access your website directly or if your mobile app needs to access resources, you must select this option. Otherwise, you may block legitimate users.
-
Clear (Forbid): If your resources, such as images, should only be embedded on your own web pages and are not intended for direct access, you can deselect this option to enhance protection.
-
-
Click OK to save the configuration.
Rate limiting
If you cannot determine whether a referer is legitimate and therefore cannot set up a referer blocklist, you can use the rate limiting feature in ESA to limit the access rate and reduce malicious traffic.
-
In the ESA console, select Websites, and click in the Actions column of the target site.
-
In the left-side navigation pane, choose .
-
On the WAF page, select the Rate Limiting Rules tab, click Create Rule, and enter the required information for the rule.
Set Hostname to your domain name and Client IP Address to Client IP. Set the Rate to an appropriate threshold for your business (we recommend 1.2 to 2 times your actual access threshold). For the action, select Apply to requests that exceed the rate limit and Block.
Select Apply to Cached Resources. Set Block Page to Default Block Page and Response Code to 403.
-
Click OK.