What is robots.txt?

A robots.txt file is a set of rules that tells a crawler which parts of your website it can access. When a crawler visits a website, it first checks for a robots.txt file in the root directory. If the file exists, the crawler follows the rules within it. If the file does not exist, the crawler assumes it has permission to access all pages that are not password-protected. If you do not need to restrict crawler access, you do not need to upload a robots.txt file.

The website backend provides a default robots.txt file. After your website is live, you can view the default file by appending /robots.txt to your domain name in a browser. The default content is shown below.

User-agent: *
Allow: /

To upload your own robots.txt file, navigate to the website backend. In the left navigation pane, click website management and then robots.txt. Then, upload your file and publish the changes.