Frequently Asked Questions

A robots.txt file is a text file that instructs search engine bots (crawlers) which pages or files on a website they are allowed or not allowed to crawl.

Yes, this generator is 100% free to use. It runs entirely in your browser and requires no registration.

The file must be uploaded to the root directory of your website. For example, if your domain is example.com, the file must be accessible at https://example.com/robots.txt.

No. It prevents crawling. If a page is blocked in robots.txt but is linked from another website, Google may still index the URL. To prevent indexing, use a noindex meta tag instead.

No. Robots.txt is a public file. Good bots will obey it, but malicious bots will ignore it. Furthermore, humans can read your robots.txt file to see exactly which URLs you are trying to hide. Always use password protection for private data.

No. Googlebot does not support the Crawl-delay directive. To change Google's crawl rate, you must use the Google Search Console settings. Bingbot and Yandex do support Crawl-delay.

If a crawler requests robots.txt and gets a 404 Not Found error, it generally assumes the site has no restrictions and will crawl the entire website.