Prevent unauthorized CDN traffic with referer blocking-CDN(CDN)-阿里云帮助中心

Background

This topic describes how to analyze access data to identify patterns of unauthorized traffic. It provides configuration methods, use cases, and recommendations for various protection solutions to help you effectively control sudden cost spikes caused by unauthorized resource usage.

Identify unauthorized traffic patterns

Basic query: offline logs

Download offline logs to view access logs for the relevant time period. Analyze the HTTP request details to identify suspicious IP addresses, User-Agents, and more. Offline logs contain limited data fields. If you want to view more data, use the real-time logs feature.

CDN logs record only basic access information, including the access time, client IP address, requested URL, User-Agent, Referer, response size, HTTP status code, and cache status. These fields cannot directly determine whether a request originates from a scraping tool or a regular browser. Although the User-Agent field can indicate the client type, it can be forged and is therefore not a reliable basis for judgment. You can analyze high-frequency IP addresses, abnormal User-Agents, and access behavior patterns to indirectly identify suspicious traffic, but you cannot confirm the specific access method or intent from the logs alone.

Use command-line tools to quickly parse the offline log files and extract information such as the top 10 IP addresses or User-Agents by request volume. For details, see Analyze offline logs.

Advanced query: operation reports and real-time logs

Important

You must customize operation reports to enable statistical analysis. If you have already configured real-time log delivery or subscribed to operation reports, you can view historical log data. Operation reports are a free, built-in feature of Alibaba Cloud CDN.
To generate real-time logs, you must first enable Log Service (SLS) and ensure logs are delivered successfully. This is a paid feature. For pricing details, see Pricing Details.
You must configure both real-time logs and operation reports in advance. If you did not configure these features before the high charges occurred, you can use only offline logs for historical analysis.

Operation reports

After customizing operation reports, you can view user access reports, including PV/UV, Regions and ISPs, Domain Name Ranking, Popular Referer Headers, Popular URLs, Popular Origin URLs, and Top Client IPs. For details, see Customize and Subscribe to Operation Reports.

For example, to view the specific files that generate CDN origin traffic from Object Storage Service (OSS), you can query the Popular Origin URLs report. This report displays the top origin-pull files for a specified domain name, status code, and date range, including traffic volume and request count.

Real-time logs

To query more log information, such as Referer and URI details, enable Log Service (SLS) to collect real-time logs. You are billed based on the number of delivered log entries after this feature is enabled.

Configure real-time log delivery for the accelerated domain name you want to analyze, as described in Configure real-time log delivery.
On the Real-time Logs page, find the project you want to analyze and click Log analysis.
On the Log Analysis page, filter the time range in the upper-right corner. Click the Raw Logs tab on the left, find the refer_domain field, and view the Referer information sorted in descending order.
Query statement example: * and refer_domain : "domain" and hit_info= HIT. In the query results, log details include refer_domain (highlighted in red), hit_info (value: HIT, highlighted in orange), return_code, method, and other fields. The refer_domain panel on the left ranks Referer domain names by proportion from high to low.

Blocking strategies

Handle empty referers

If you experience a large volume of traffic from requests with an empty referer, you can use Edge Security Accelerator (ESA) to manage it.

If your business allows access from an empty referer, you can configure a JavaScript challenge for empty referer requests to block unauthorized traffic. If your business does not allow empty referer access, you can block these requests directly.

JavaScript challenge

In the ESA console, select Websites. In the Website column, click the target website.
In the left-side navigation pane, choose Security > WAF.
Click the Custom Rules tab, and on the Custom Rules tab, click Create Rule.
- Enter a Rule Name.
- In the If requests match... area, configure the match conditions. For more information, see Rule expression syntax.
- In the Then execute... area, specify an action for matching requests. For more information, see Actions.
Click OK.

For the match condition, click Edit Expression and enter the following expression: (exists(http.referer) and len(http.referer) lt 1 and http.host eq "xxxxxx.aliyun.com") or (not exists(http.referer) and http.host eq "xxxxxx.aliyun.com") Replace xxxxxx.aliyun.com with your actual domain name.
Set the action to JavaScript challenge.

Block empty referers

Use case 1: Business does not allow any empty Referer requests. Configure a WAF custom rule in ESA to block all empty Referer requests. Configuration path: Site > Security Protection > WAF > Custom Rules.
Expression: (exists(http.referer) and len(http.referer) lt 1) or (not exists(http.referer))
When a request matches the rule, set the action to Block, the response code to 403, and select the default block page.
Use case 2: Business needs to allow some empty Referer requests, such as homepage access. Configure WAF rules to block empty Referer requests, and add whitelist rules to allow specific paths, such as the home page. Proceed as follows:
1. Configure a custom rule to block empty Referer. Configuration path: Site > Security Protection > WAF > Custom Rules.
  Use the same expression as in use case 1: (exists(http.referer) and len(http.referer) lt 1) or (not exists(http.referer)). When a request matches the rule, set the action to Block, the response code to 403, and select the default block page.
2. Configure a whitelist rule to allow specific paths, such as the home page. Configuration path: Site > Security Protection > WAF > Whitelist Rules.
  Expression: (http.host in {"your hostname"} and http.request.uri in {"/"}). On the Add Whitelist Rule page, set the match condition where Hostname equals your target domain (for example, your hostname), and add a second condition with an And operator where URI equals /. In the Then skip section, select All rules, and then click OK.

H5, miniapp, APP, and client scenario

This scenario extends the APP and client program scenario by also excluding miniapp requests. First, exclude miniapp and APP requests. Then, set allowed Referers for the remaining traffic.

Configure the rules engine to separate APP, client program, and miniapp UAs from other UAs. This example uses the Alipay mini program and java-1.1.
Configure a Referer hotlink protection whitelist. Allow access only from specified Referers.

H5, APP, and client program scenario

Unlike browsers, APPs and client programs do not follow standard HTTP Referer behavior. They construct requests directly, so most requests lack a Referer. If you block all empty-Referer requests, you might disrupt normal APP traffic. Apply Referer-based blocking only to web traffic, not to APP or client requests.

For example, use a distinctive user agent (UA) string that differs from typical browser UAs. The following example uses the client program UA java-1.1.

Configure the rules engine to separate APP and client program UAs from other UAs.
Configure a Referer hotlink protection whitelist. Allow access only from specified Referers for web traffic.

Miniapp, APP, and client scenario

Miniapp UAs are relatively fixed but differ across vendors. Common examples include the following:

WeChat mini program: The key identifier in the UA is MicroMessenger/x.x.x.
Alipay mini program: The key identifier in the UA is AlipayClient/x.x.x+ Nebula.
Baidu mini program: The key identifiers in the UA are BaiduApp/x.x.x or swan.
ByteDance (Douyin): The key identifier in the UA is BytedanceWebview/ Douyin.

First, identify the primary miniapp type your business uses, and then configure the settings accordingly. The rules engine determines the appropriate solution.

Configure the rules engine to separate APP, client program, and miniapp UAs from other UAs. This example uses the Alipay mini program and java-1.1.
Configure a Referer hotlink protection whitelist. Allow access only from specified Referers. Because miniapps and APPs rarely send real Referers, create a fake Referer that does not exist and add it to the whitelist. Apply this rule globally.

APP and client scenario

In a pure APP or client program environment, high-frequency traffic from the application's own UA may indicate an issue within the application. If the UA in the traffic does not match the application's UA, apply the same solution as described in the H5, APP, and client program scenario.

Configure the rules engine to separate APP and client program UAs from other UAs.
Configure a Referer hotlink protection whitelist. Allow access only from specified Referers. Because real Referers are absent, specify a placeholder Referer value that no real client sends and add it to the whitelist. Apply this rule globally.

Large file transfer scenario

Large file transfers are slow, infrequent, and lack predictable access patterns. This makes them difficult to protect. Even low-frequency access can consume large amounts of bandwidth.

Solution 1: Per-request rate limiting

Use per-request rate limiting to restrict requests from non-APP or non-miniapp clients.

Configure the rules engine to detect client features. Use the UA string for detection. This example uses the Alipay mini program and java-1.1.
Set the per-request rate limit and associate it with the rule that you created.

Solution 2: Authentication-based protection

If you can modify your server to add authentication, you can protect large files without limiting download speed. Choose between URL authentication and remote authentication.

URL authentication

This method requires modifying the large file URL on your server. Use the authentication logic of Alibaba Cloud CDN to generate time-limited URLs. For more information, see type A signing. After you update your server, configure the matching authentication settings in the CDN console.

Remote authentication

This method does not require changes to the request URL. Instead, deploy a dedicated authentication server that checks client features to decide whether to allow access.

Deploy a remote authentication server. Design the authentication logic to limit requests per client. For example, count requests per client IP over a 24-hour period and block access if the count exceeds your threshold.
In the CDN console, enter the authentication server address. Pass the client IP and any other required client information as parameters to the server.

Block malicious referers

By analyzing unauthorized traffic patterns, you can identify illegitimate referers. If you find malicious referers, use a referer blocklist to block them.

Log on to the CDN console.
In the left navigation pane, click Domain Names.
On the Domain Names page, find the target domain name and click Manage in the Actions column.
In the domain's navigation pane, click Access Control.
On the Referer Black/Whitelist tab, click Modify.
Set Type to Blacklist. In the Rules field, enter the list of malicious referers. Select Ignore Scheme.
Decide whether to select Allow Empty Referer based on your business needs.
- Select (Allow): If your users access your website directly or if your mobile app needs to access resources, you must select this option. Otherwise, you may block legitimate users.
- Clear (Forbid): If your resources, such as images, should only be embedded on your own web pages and are not intended for direct access, you can deselect this option to enhance protection.
Click OK to save the configuration.

Rate limiting

If you cannot determine whether a referer is legitimate and therefore cannot set up a referer blocklist, you can use the rate limiting feature in ESA to limit the access rate and reduce malicious traffic.

In the ESA console, select Websites, and click WAF in the Actions column of the target site.
In the left-side navigation pane, choose Security > WAF.
On the WAF page, select the Rate Limiting Rules tab, click Create Rule, and then enter the rule information based on the on-screen instructions.
Set Hostname to your domain name and Client IP Address to Client IP. Set the Rate to an appropriate threshold for your business (we recommend 1.2 to 2 times your actual access threshold). For the action, select Apply to Matched Requests and Block.
Select Apply to Cache. Set the block page to Default Error Page and Response Code to 403.
Click OK.