Allow crawlers from authorized search engines such as Google, Bing, Baidu, Sogou, and Yandex to access your domain.
Prerequisites
-
A WAF instance is activated and meets these requirements:
-
Subscription instance: The Bot Management module is enabled.
-
Pay-as-you-go instance: The Authorized Crawler feature of the Bot Management module is enabled in the Bills and Plans. Billing & Package Center (Pay-As-You-Go 2.0).
-
-
Your website has been added to WAF. For more information, see Tutorials.
How it works
Allowed Crawlers rules use the Alibaba Cloud crawler library to automatically allow legitimate crawler requests. The library is continuously updated by analyzing Alibaba Cloud network traffic and dynamically updates IP addresses for major search engines, including Google, Baidu, Sogou, Bing, and Yandex.
After you enable Allowed Crawlers rules, requests from authorized search engine IP addresses automatically bypass Bot Management detection.
You can also use Access Control/Throttling rules to filter requests from allowed crawler IP addresses. Configure a custom protection policy.
Procedure
-
Log on to the Web Application Firewall console.
-
In the top menu bar, select the resource group and region for your Web Application Firewall instance (Chinese Mainland or Outside Chinese Mainland).
-
In the left navigation pane, choose .
-
On the Website Protection page, switch to the domain name to configure.
-
Click the Bot Management tab. In the Authorized Crawler section, turn on the Status switch and click Configure Now.
-
In the Authorized Crawler rule list, find the target crawler by Intelligence Name and turn on its Status switch.
By default, configure individual rules for specific search engines (Google, Bing, Baidu, Sogou, and Yandex), or enable the Authorized Search Engine Whitelist rule to allow all supported crawlers.