Robots.txt
Updated at:
Copy as MD
Robots.txt is a text file that controls which parts of your site web crawlers can access. This topic describes how to add a Robots.txt file.
Background information
When a search spider visits a site, it first checks for a Robots.txt file in the site's root directory. If the file exists, the spider follows the rules in the file to determine which pages to crawl. If the file does not exist, the spider can access all pages on the site that are not protected by a security token.
Procedure
In the navigation pane on the left, click .
Upload the Robots.txt file. For more information about the Robots.txt format, see Site Management FAQ.
Is this page helpful?