// definition
ClaudeBot
Anthropic's training crawler for Claude. You can opt out of model training with ClaudeBot while still allowing Claude-SearchBot to fetch you for answers.
What it is
ClaudeBot is the web crawler run by Anthropic, the company behind the Claude models. Its job is collecting data to train future versions of Claude, and it respects robots.txt, so you can allow or block it with a normal user-agent rule.
Anthropic runs two other crawlers you should know about. Claude-SearchBot handles retrieval: it fetches pages so Claude can cite live sources in an answer. Claude-User covers user-initiated fetches, when someone asks Claude to read a specific page. These are controlled independently, which matters more than it sounds.
Like most AI crawlers, ClaudeBot reads raw HTML and does not execute JavaScript.
Why it matters
Because training and retrieval are separate crawlers, you get a real choice. You can block ClaudeBot to keep your content out of Claude's training data while still letting Claude-SearchBot reach you, so Claude can cite you in answers right now. Blocking the training crawler does not remove you from search.
The JavaScript point bites people. Vercel and MERJ found that the major AI crawlers fetch raw HTML and never run JS (the rise of the AI crawler). If your content only appears after client-side rendering, ClaudeBot and friends see an empty page. Server-side rendering fixes it.
This split mirrors OpenAI's setup, which we break down under GPTBot. For the full map, see how AI crawlers work.
// related terms
// next step
See how legible your site is to AI.
Free first audit. No credit card. Your Legibility Score in under two minutes.