Robots.txt is a plain text file that tells web crawlers which parts of your site they can access. This topic describes how to add a Robots.txt file.
Background information
When a search engine crawler visits a site, it first checks for a Robots.txt file in the site's root directory. If the file exists, the crawler uses its contents to determine which pages to access. If the file does not exist, the crawler can access all pages on the site that are not protected by a security token. If you want crawlers to access all pages, do not create this file.
Procedure
In the navigation pane on the left, click .
Upload the Robots.txt file. For more information, see Block search engines from scraping site content using robots.txt.
该文章对您有帮助吗？