// free tool
robots.txt generator
robots.txt tells crawlers which parts of your site they can access. You can allow or block AI crawlers like GPTBot and ClaudeBot exactly the same way as Googlebot — one directive per agent. Configure your rules below and download the file.
Crawlers
*
Googlebot
Bingbot
AI crawlers
GPTBot, ClaudeBot, etc.GPTBot
ClaudeBot
PerplexityBot
Google-Extended
CCBot
Disallow paths
Applied to allowed crawlers only.
Preview
User-agent: * Allow: / Disallow: /admin User-agent: Googlebot Allow: / Disallow: /admin User-agent: Bingbot Allow: / Disallow: /admin User-agent: GPTBot Allow: / Disallow: /admin User-agent: ClaudeBot Allow: / Disallow: /admin User-agent: PerplexityBot Allow: / Disallow: /admin User-agent: Google-Extended Allow: / Disallow: /admin User-agent: CCBot Allow: / Disallow: /admin
// place robots.txt at your domain root: example.com/robots.txt