Use bot management to control AI crawlers
For requests from AI crawlers, AI agents, and AI search bots, you can use the AI crawler control feature in Web Application Firewall (WAF) bot management to view request overviews and detailed data, and take action on traffic with a single click. These automated programs frequently scrape web content for model training, building knowledge bases, and supporting AI applications. This feature helps you accurately identify and manage such traffic to protect your intellectual property and core business data, while preventing excessive consumption of server bandwidth.
Key concepts
AI crawler
Core purpose: To batch-scrape site content for large language model training or to build a corpus for Retrieval-Augmented Generation (RAG).
Behavioral characteristics: Performs large-scale, exhaustive data scraping. These programs typically declare a dedicated User-Agent in the request header and state that they follow the robots.txt protocol.
Difference from traditional crawlers: The scraped content is primarily used for model training or corpus building and typically does not drive direct traffic to the site.
Typical examples: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, CCBot (Common Crawl), and Bytespider.
AI search
Core purpose: To fetch specific content from a site in response to a user's real-time query and use the content as a cited source for a generated answer.
Behavioral characteristics: Driven by a single user query, it performs real-time, on-demand, single-page access rather than background bulk scraping. This type of access usually includes source links in the generated answer, providing exposure and referral value to the site.
Typical examples: OAI-SearchBot (OpenAI), PerplexityBot (Perplexity), and Gemini-Deep-Research (Google).
AI agent
Core purpose: To perform specific business operations on a site on behalf of a real user, such as querying information, comparing prices, filling out forms, logging in, or submitting orders.
Behavioral characteristics: Engages in interactive behavior with clear intent. In addition to reading content, it executes complete business workflows, including clicks and form submissions. These programs often operate within a real browser environment, and their access fingerprints closely mimic those of human users. This is the most difficult type of access to identify and the one most likely to pose business security risks.
Typical examples: ChatGPT-User (OpenAI), Claude-User (Anthropic), and various AI agents based on Computer-Use or browser automation technologies.
Procedure
Meet the following prerequisites:
A protected object has been added: You must add your web services to WAF. If you have not yet added your services, see Onboarding overview.
Bot management is enabled: Subscription users must enable bot management (this feature is not supported by the Basic subscription edition). For details, see Enable bot management.
Log on to the Web Application Firewall 3.0 console. In the top navigation bar, select the resource group and region (Chinese Mainland or Outside Chinese Mainland) of your WAF instance. In the left-side navigation pane, choose .
AI crawler request overview
On the AI Crawler Management page, the system displays Total AI Crawler Requests, Detected AI Crawlers, and Blocked AI Crawler Requests. A detailed data list is provided at the bottom of the page. You can use the search bar at the top to filter by Time, Protected object, Path, and AI category.
Fields
Name: The specific name of the AI crawler or agent.
Category: The AI type, which can be AI crawler, AI agent, or AI search.
Requests and Blocked Requests: The total number of requests from the AI and the number of requests blocked by WAF during the specified period.
Trend Graph: A visualization of request volume over time.
Top Interfaces: The access paths with the highest request volume (for example,
/index.php).Protected Object: The protected object added to WAF.
Manage AI crawler requests
By default, WAF does not have any rules for AI crawler requests. If this behavior does not meet your business requirements, you can manually manage specific AI requests. Perform the following steps:
On the AI Crawler Management page, find the target AI name in the list at the bottom.
In the Actions column, click Log or Block.
Log: Logs requests but does not block traffic.
Block: Logs requests and blocks traffic.
WAF automatically generates a corresponding Advanced Custom Rules template and rule based on your selection. You can view them on the Advanced Custom Rules page. For more information, see Configure advanced custom rules for bot management.
Configuration recommendations
Apply different control policies to different types of AI traffic. Avoid using a uniform mechanism to block or allow all traffic.
I. AI crawler control policy
Set authorization and basic controls: First, evaluate whether to allow your site content to be used for model training. If you deny authorization, block or apply rate limiting to training crawlers. If you grant authorization, apply rate limiting to prevent high-frequency crawling from affecting origin server performance.
Implement tiered control: Manage content differently based on its value. Apply strict controls to core pages, such as main content and product detail pages. Relax restrictions for general content, such as public introductory pages.
II. AI search control policy
Allow by default and monitor: Most sites want to gain exposure and citation value from AI search. Allow this traffic by default, and also configure rate and traffic monitoring.
Use fine-grained permission control: If you have concerns about content copyright or citation scope, set access restrictions for specific directories or pages. This ensures public content can be cited correctly while preventing sensitive content leaks.
III. AI agent control policy
Integrate with business security risk control: This traffic poses the highest risk. Do not simply block it. Instead, integrate it with bot management for Web Protection/App Protection and account security protection.
Strengthen protection for key operations: For key operations such as logins, placing orders, claiming coupons, and submitting forms, add layers of account security protection, rate limiting, and CAPTCHA. This helps prevent automated fraud, such as bulk registration, inventory hoarding, and malicious arbitrage. It also ensures that the user experience for legitimate human users is not affected.
IV. Policy implementation and dynamic adjustment
Combine the preceding policies as needed for your business. First, use the traffic analysis and statistical reports from the AI crawler control system. Continuously monitor the composition and trends of different AI traffic types. Based on this information, gradually adjust your allow, rate limiting, and block rules. This helps you find the best balance between content protection, business security, and traffic value.