Robots.txt Examples & Templates
Common configurations and templates you can copy for your own website.
1. Allow all crawling (Default)
User-agent: *
Disallow:2. Block entire site from all crawlers
User-agent: *
Disallow: /3. Block a specific folder
User-agent: *
Disallow: /admin/