Robots.txt
更新时间：
复制 MD 格式
A Robots.txt file instructs web crawlers which pages of a site they can access. This topic describes how to add a Robots.txt file.
Background information
When a search engine crawler visits a site, it first checks for a Robots.txt file in the site's root directory. If the file exists, the crawler determines which pages it can access based on the file's content. If the file does not exist, the crawler can access all pages on the site that are not protected by a security token. If you want crawlers to access all pages, do not create this file.
Procedure
- In the navigation pane on the left of the site console, click .
- Upload the Robots.txt file. For more information about the Robots.txt syntax, see Block search engines from scraping website content using robots.txt.
该文章对您有帮助吗？