Robots.txt Configuration

Configure your crawler directives below. The file generates instantly in real-time.

Default Rule (User-agent: *)

Allow All Access Disallow Entire Site

Allow Specific Directories (One per line)

Disallow Specific Directories (One per line)

Crawl Delay (Seconds) Supported by Bing/Yandex to prevent server overload.

Host Directive (Optional) Supported by Yandex for canonical domain definition.

XML Sitemaps (One URL per line)

Googlebot (Google Search)

Bingbot (Bing Search)

Add specific Allow/Disallow directives for bots like Baiduspider, Yandex, AhrefsBot, etc.

Characters: 0 Lines: 0