Configure bot management

Set website anti-crawler rules

If your users access web pages or H5 pages (including the H5 pages that are used in apps) through a browser, set anti-crawler rules for browser-based access to protect your services against crawler risks in a more targeted manner.

1. Log on to the DCDN console.

2. In the left-side navigation pane, choose WAF > Protection Policies.

3. On the Protection Policies page, click Create Policy.

4. On the Create Policy page, configure the protection settings.

Configuration module	Configuration item	Description
Policy Information	Policy Type	Select Bot Management.
	Policy Name	The custom name of the policy. The name can contain Chinese characters, letters (uppercase and lowercase), digits (0 to 9), and underscores (_), and can be up to 64 characters in length.
Global Configurations	Service Type	Select Websites to protect web pages or H5 pages that are accessed through a browser, including content that is rendered as pure H5 pages in apps.
	Web SDK Integration	Automatic integration (recommended): Uses a JavaScript-based web SDK to improve protection in web browser scenarios and to avoid some compatibility issues. After you enable automatic integration, WAF automatically references the SDK in the HTML pages of the protected object. The SDK collects browser information, specific attack and defense probes, and operation behavior, which do not involve sensitive personal information. WAF then identifies and blocks request risks based on the collected information. Manual integration: If automatic integration does not apply to your current environment, use manual integration. For more information, see Integrate an SDK for a web application.
	Traffic Characteristics	Add the HTTP request fields and rules of the target traffic. These fields are the content that is generated in the HTTP request message for the protected service scenario when the protected target is accessed. For more information about the fields, see Match conditions.
Legitimate Bot Management	Spider Whitelist	After you turn on this switch, the crawler IP addresses of mainstream search engines are supported and dynamically updated. The supported search engines are Google, Baidu, Sogou, Bing, 360, and Yandex. After you enable the rule, legitimate crawler IP addresses from these search engines are allowed without protection detection by the bot management module.
Bot Characteristic Detection	Script-based Bot Block (JavaScript)	After you turn on this switch, a JavaScript check is performed on the clients that access the anti-crawler protected target. Traffic from non-browser tools that do not support JavaScript checks is filtered out to block simple script-based attacks.
	Advanced Bot Defense (Dynamic Token-based Authentication)	After you turn on this switch, the signature of each request is verified, and requests that fail signature verification are blocked. You can select signature verification failure (required, which indicates that the request carries no signature or an invalid signature), invalid signature timestamp, and WebDriver attacks.
Bot Behavior Detection	AI Intelligent Protection	After you turn on this switch, the anti-crawler rules use intelligent protection engines to analyze access traffic and learn from it automatically, and then generate targeted protection rules or blacklists. Monitor: The anti-crawler rules allow the matched traffic and record it in security reports. Slider CAPTCHA: The client must complete a slider CAPTCHA before it can continue to access the protected target.
Custom Throttling	IP Address Throttling (Default)	Specifies that when the number of requests from the same IP address exceeds the specified threshold within the statistical interval, a throttling action is performed on the requests from that IP address. The action can be slider CAPTCHA, block, or monitor. You also specify the duration of the throttling action. You can set up to three conditions. For more information, see custom rule parameters.
	Custom Session Throttling	You can set the session type and define custom session throttling conditions. These conditions specify that when the number of requests from the same session exceeds the specified threshold within the statistical interval, a throttling action is performed on that session. The action can be slider CAPTCHA, block, or monitor. You also specify the duration of the action. You can set up to three conditions. For more information, see custom rule parameters.
Bot Threat Intelligence	Bot Threat Intelligence Library	Contains the IP addresses of attack sources that have performed malicious crawling against multiple Alibaba Cloud users multiple times over a period of time. You can set the bot threat intelligence library to monitor or slider CAPTCHA.
	Data Center Blacklist	After you turn on this switch, the selected IP address libraries are blocked. If you use source IP addresses from a public cloud or a data center to access your services, add known legitimate calls to the whitelist, such as payment callbacks from Alipay or WeChat and monitoring programs. The data center blacklist supports the following IP address libraries: Alibaba Cloud, 21Vianet, Meituan Cloud, Tencent Cloud, and Others. You can set the data center blacklist to monitor, slider CAPTCHA, or block.
	Fake Spider Blocking	After you turn on this switch, the User-Agent of all search engines in legitimate bot management is blocked. Legitimate client IP addresses of the search engines for which the whitelist is enabled are allowed.
Protected Domain Names	Select Association Mode	Add and replace the original associated policy: Unbinds the associated policy and replaces it with the current policy. Add and keep the original associated policy: The current policy and the bound policy coexist and do not affect each other.
	Protected Domain Names	Select the domain names that you want to add to the current mitigation policy. Note A protected domain name can be associated with only one mitigation policy of the same policy type. If the domain name is already associated with another policy of the same type, the policy information of the domain name is replaced with the current policy after you apply the current policy to the domain name. You cannot configure bot management for DCDN domain names for which WebSocket is enabled, because WebSocket content is encrypted and attack characteristics cannot be identified.

5. Click Create Policy.

   The new protection policy is enabled by default.

Set app anti-crawler rules

If you use an app that is natively developed for iOS or Android (excluding the H5 pages that are used in the app), set app anti-crawler rules to protect your services against crawler risks in a more targeted manner.

1. Log on to the DCDN console.

2. In the left-side navigation pane, choose WAF > Protection Policies.

3. On the Protection Policies page, click Create Policy.

4. On the Create Policy page, configure the protection settings.

   Configuration module	Configuration item	Description
   Policy Information	Policy Type	Select Bot Management.
   	Policy Name	The custom name of the policy. The name can contain Chinese characters, letters (uppercase and lowercase), digits (0 to 9), and underscores (_), and can be up to 64 characters in length.
   Global Configurations	Service Type	Select APP to protect apps that are natively developed for iOS or Android, excluding the H5 pages that are used in the app.
   	Web SDK Integration	Uses the SDK for native apps (Android or iOS) to improve protection in app scenarios. After the SDK is integrated, it collects the risk characteristics of the client and generates a security signature that is attached to requests. WAF identifies and blocks request risks based on the signature characteristics. Click Get and copy appkey, and then fill in the information to request the SDK package. For more information, see Integrate the SDK into Android apps or Integrate the SDK into iOS apps.
   	Traffic Characteristics	Add the HTTP request fields and rules of the target traffic. These fields are the content that is generated in the HTTP request message for the protected service scenario when the protected target is accessed. For more information about the fields, see Match conditions. You can add up to five conditions.
   Bot Characteristic Detection	Invalid App Signature	Invalid App Signature is selected by default and cannot be disabled. The anti-crawler rules detect requests that carry no signature or an invalid signature after the SDK is integrated into the app.
   	Abnormal Device Behavior	After you enable this item, the anti-crawler rules detect and control the requests that are sent from devices with abnormal characteristics. The abnormal device characteristics include: Expired Signature: Enabled by default. Indicates that the timestamp of the device request has expired. Using Simulator: Indicates that an emulator is used on the device. Using Proxy: Indicates that a proxy service is used on the device. Rooted Device: Indicates that root permissions are enabled on the device. Debugging Mode: Indicates that debugging mode is enabled on the device. Hooking: Indicates that a hook program exists on the device. Multiboxing: Indicates that multiple processes of the protected app are open on the device at the same time. Simulated Execution: Indicates that operations that simulate user behavior exist on the device. Script Tools: Indicates that automatically run scripts exist on the device.
   	Custom Signature Field	Select a field name to define a custom signature field in header, Parameter, or cookie. If the object to be signed is unusual, for example, the body is oversized, empty, or specially encoded, you can process the signature content by using a method such as a hash and place the result in the custom signature field. WAF then verifies the signature based on the content of this field.
   	Action	You can set the rule to monitor or block based on your requirements: Monitor: Triggers an alert but does not block the request. Block: Blocks the attack request directly.
   	Secondary Packaging Detection	After you enable this item, app requests whose package name and package signature are not in the whitelist of valid package names and package signatures are considered requests from a repackaged app. You can specify valid version information: Specify a valid package name: Specifies the name of a valid app package. For example, example.aliyundoc.com. Package signature: Contact Alibaba Cloud security engineers to obtain the package signature. If you do not need to verify the package signature of the app, leave the package signature blank. WAF then verifies only the specified valid app package name. Note The package signature is not the app certificate signature. You can add valid versions for both iOS and Android packages, up to five in total, and package names cannot be duplicated. You can set the rule to monitor or block based on your requirements.
   Throttling	IP Address Throttling (Default)	Specifies that when the number of requests from the same IP address exceeds the specified threshold within the statistical interval, a throttling action of block or monitor is performed on the requests from that IP address. You also specify the duration of the throttling action. You can set up to three conditions. For more information, see custom rule parameters.
   	Device Throttling	You can set device throttling conditions to specify that when the number of requests from the same device exceeds the specified threshold within the statistical interval, a throttling action of block or monitor is performed on the requests from that device. You also specify the duration of the throttling action. You can set up to three conditions. For more information, see custom rule parameters.
   	Custom Session Throttling	You can set the session type and define custom session throttling conditions. These conditions specify that when the number of requests from the same session exceeds the specified threshold within the statistical interval, a throttling action of block or monitor is performed on that session. You also specify the duration of the action. You can set up to three conditions. For more information, see custom rule parameters.
   Bot Threat Intelligence	Bot Threat Intelligence Library	Contains the IP addresses of attack sources that have performed malicious crawling against multiple Alibaba Cloud users multiple times over a period of time. You can set the bot threat intelligence library to monitor or slider CAPTCHA.
   	Data Center Blacklist	After you turn on this switch, the selected IP address libraries are blocked. If you use source IP addresses from a public cloud or a data center to access your services, add known legitimate calls to the whitelist, such as payment callbacks from Alipay or WeChat and monitoring programs. The data center blacklist supports the following IP address libraries: Alibaba Cloud, 21Vianet, Meituan Cloud, Tencent Cloud, and Others. You can set the data center blacklist to monitor, slider CAPTCHA, or block.
   Protected Domain Names	Select Association Mode	Add and replace the original associated policy: Unbinds the associated policy and replaces it with the current policy. Add and keep the original associated policy: The current policy and the bound policy coexist and do not affect each other.
   	Protected Domain Names	Select the domain names that you want to add to the current mitigation policy. Note A protected domain name can be associated with only one mitigation policy of the same policy type. If the domain name is already associated with another policy of the same type, the policy information of the domain name is replaced with the current policy after you apply the current policy to the domain name.

5. Click Create Policy.

   The new protection policy is enabled by default.

Related APIs

• CreateDcdnWafPolicy — Creates a custom protection policy.

• BatchCreateDcdnWafRules — Creates multiple custom rules in a batch.

• DescribeDcdnWafPolicies — Queries custom protection policies.

• DescribeDcdnWafDefaultRules — Queries the default rule set.

• DescribeDcdnWafPolicies — Queries custom protection policies.

• DescribeDcdnWafPolicyValidDomains — Queries the valid domains for a custom protection policy.