Robots.txt generator

Create a robots.txt file with crawler rules and an optional sitemap URL.

Text Stateless

Crawler directives do not control access to private pages.

Quick guide

Try an example

You enter

Crawler name
*
Disallowed paths
/drafts/
Sitemap URL
https://example.com/sitemap.xml

You get

User-agent: *
Disallow: /drafts/

Sitemap: https://example.com/sitemap.xml
How it works and limits

Enter one path per line beginning with /. Leave the paths blank to allow all paths for the selected crawler.