ClaudeBot is Anthropic's crawler that collects public web content to train Claude models. Site owners can block it in robots.txt with the ClaudeBot token.
What is ClaudeBot?
ClaudeBot is the web crawler Anthropic uses to collect publicly available web content that may be used to train its Claude models. It identifies itself with the ClaudeBot user agent and honors robots.txt. Disallowing it signals to Anthropic that your site’s future content should be excluded from its training datasets. It is one of three Anthropic agents.
The name is often used loosely for “Anthropic’s bot”, but that blurs an important distinction. Anthropic separates training, search indexing and user-triggered retrieval into different agents, and each robots.txt decision has a different effect.
How it works
Anthropic’s support documentation lists three agents:
| Agent | Purpose | Effect of blocking |
|---|---|---|
| ClaudeBot | Collects content for model training | Future content excluded from training |
| Claude-SearchBot | Indexes the web to improve search results | May reduce visibility in Claude search answers |
| Claude-User | Fetches pages when a user asks | May reduce visibility in user-directed search |
Other documented behavior:
- Anthropic’s bots honor robots.txt “do not crawl” directives. Rules must be set on each subdomain.
- Anthropic supports the non-standard
Crawl-delaydirective to slow crawling. - Anthropic publishes its crawler IPs for identification, but says blocking by IP is not a reliable way to stop crawling; use robots.txt.
- Anthropic says it also recognizes a machine-readable “no AI training” signal based on an IETF draft, with rules that name ClaudeBot taking priority.
From SEO to AEO and GEO
There is no classic SEO equivalent to a training crawler. Search engines crawled to build an index you could rank in; ClaudeBot crawls to improve a model that may later answer questions about your category from memory. That is a different kind of visibility: not a ranking, but whether the model knows your product exists and describes it correctly.
What carries over from SEO is the operational side: robots.txt, log analysis, server-rendered content and avoiding accidental CDN blocks. What changed is that “allow or block” is now a policy decision with trade-offs. Allowing training may help models describe you accurately; blocking protects proprietary content but can leave models relying on third-party descriptions of your brand.
For a B2B SaaS company, that trade-off usually favors allowing ClaudeBot on public marketing pages and docs, because accurate product facts in training data reduce the risk of outdated or wrong descriptions. The exceptions are content you license, sell or keep behind a login.
What to do about it
- List all three Anthropic agents in your robots.txt policy and decide on each separately.
- Keep Claude-SearchBot and Claude-User open on public pages if you want Claude to cite you in web answers.
- Decide on ClaudeBot by content type. Marketing pages and public docs are usually fine to allow; licensed research or paid content may not be.
- Use Crawl-delay if ClaudeBot load is the concern, rather than blocking outright.
- Check logs and the edge. Confirm 200 responses and that bot-management rules are not challenging Anthropic’s agents.
- Re-check the support article periodically; agent names have changed before.
Common mistakes
- Writing rules only for legacy names such as anthropic-ai or Claude-Web.
- Blocking ClaudeBot and assuming Claude search is unaffected without also checking Claude-SearchBot and Claude-User rules.
- Forgetting subdomains. docs. and blog. subdomains need their own robots.txt.
- Blocking by IP, which Anthropic says is unreliable.
Related terms
Sources
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic support
- Robots.txt introduction, Google Search Central (syntax reference)
Frequently asked questions
Does blocking ClaudeBot stop Claude from citing my site?
Not directly. Anthropic describes ClaudeBot as its training crawler. Its documentation says blocking Claude-SearchBot or Claude-User is what may reduce visibility and accuracy in user search results. Training access still shapes how Claude describes your brand when it answers without searching, so blocking ClaudeBot has indirect effects.
Does ClaudeBot support Crawl-delay?
Yes. Anthropic says it supports the non-standard Crawl-delay extension in robots.txt, for example a ClaudeBot group with Crawl-delay: 1. Note that Google ignores Crawl-delay, so this is an Anthropic-specific way to reduce crawl rate without blocking.
What about older user agents like anthropic-ai and Claude-Web?
Many robots.txt templates still list those older names. Anthropic's current support article documents ClaudeBot, Claude-User and Claude-SearchBot, so write rules for those three. Keeping legacy lines is harmless, but they should not be your only Anthropic rules.
Guides that cover ClaudeBot
-
LLMs.txt vs Robots.txtLLMs.txt vs Robots.txt: How to Optimize Your Website for AI Crawlers & Answer Engines
LLMs.txt vs Robots.txt explained: what each file does, who reads it, and how to implement both to optimize your site for AI crawlers and answer engines
