Allow AI search and retrieval crawlers if you want to appear in AI answers, and decide on training crawlers separately. For most B2B companies the right default is to allow both: OAI-SearchBot, PerplexityBot and Claude-SearchBot get you cited in live answers, while GPTBot, ClaudeBot and Google-Extended shape what models know about your brand. Block training crawlers only if your content itself is the product.
Training crawlers vs search crawlers
The most common mistake is treating all AI bots the same. Providers now separate crawlers by purpose.
| User agent | Provider | Purpose | Block it and |
|---|---|---|---|
| GPTBot | OpenAI | Collects content for model training | Content not used for future training |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search | Not shown in ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetches pages when a user or GPT requests them | User-triggered visits blocked where honored |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers | Not surfaced in Perplexity results |
| ClaudeBot | Anthropic | Collects content for model training | Content not used for future training |
| Claude-SearchBot | Anthropic | Indexes pages for Claude’s search results | Less visible in Claude search answers |
| Google-Extended | Token controlling use in Gemini training and grounding | No effect on Google Search or AI Overviews |
Sources: OpenAI bots, Perplexity bots, Anthropic crawler policy, Google common crawlers. Providers update these pages, so check them before changing rules.
Two clarifications matter:
- Google AI Overviews and AI Mode use Googlebot, the same crawler as Search. Google-Extended is a control token, not a separate crawler, and Google says it does not affect Search inclusion or ranking.
- User-initiated fetchers (ChatGPT-User, Perplexity-User, Claude-User) act on behalf of a person. Providers note that robots.txt may not apply in the same way to these requests.
How to decide
flowchart TD
A[Do you want to be cited in AI answers?] -- Yes --> B[Allow search crawlers: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot]
A -- No --> C[Block search crawlers, accept lost visibility]
B --> D[Is your content itself the product, e.g. paid research or licensable data?]
D -- No --> E[Allow training crawlers too]
D -- Yes --> F[Block training crawlers: GPTBot, ClaudeBot, Google-Extended]
Allow both if
- You are a B2B SaaS, services or product company and your content exists to attract buyers
- You want models to describe your brand, category and features accurately
- Your public pages contain nothing you would not want summarized
Block training, allow search if
- You publish paid research, premium data or content you license
- You want AI search referrals but do not want your content in training sets
Block both if
- Content is private, gated or legally sensitive, in which case it should not be publicly accessible in the first place
robots.txt examples
Allow everything (typical B2B marketing site):
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: *
Allow: /
Disallow: /app/Allow AI search, block AI training:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Block training on one premium section only:
User-agent: GPTBot
Disallow: /research/
User-agent: ClaudeBot
Disallow: /research/Keep your sitemap line and any existing rules for Googlebot and Bingbot. For more context, see our glossary entry on the robots.txt file and llms.txt vs robots.txt.
Do not forget the CDN and firewall
robots.txt is a request, not an enforcement mechanism. Many sites lose AI visibility not because of robots.txt but because a CDN or WAF blocks AI user agents. Some CDNs, including Cloudflare, have introduced settings that block AI crawlers by default for new sites. Check:
- Bot management rules in your CDN or WAF
- Rate limits that return 403 or 429 errors to crawlers
- Server logs showing successful (200) responses to OAI-SearchBot, PerplexityBot and Claude-SearchBot
- That main content is server-rendered, since many AI crawlers do not run JavaScript reliably
Verify that crawlers are genuine
User agent strings are easy to fake, so scrapers sometimes pretend to be GPTBot or PerplexityBot. Before writing firewall exceptions, verify the traffic:
- OpenAI and Perplexity publish the IP ranges their crawlers use on their bot documentation pages
- Google recommends verifying Googlebot with a reverse DNS lookup followed by a forward lookup
- Many CDNs maintain verified-bot lists you can allow by category instead of by user agent
Allow verified search and training crawlers you have chosen to permit, and rate-limit or block unverified traffic claiming to be them. This protects server resources without cutting off the crawlers that drive AI visibility.
Common mistakes
- Blocking GPTBot and assuming that removes you from ChatGPT search, or blocking OAI-SearchBot when you only meant to stop training
- Blocking Google-Extended to “opt out of AI Overviews”, which it does not do
- Copying a “block all AI bots” list from a publisher whose business model differs from yours
- Forgetting the CDN layer
Summary
Separate the two questions. Search crawlers decide whether you can be cited today. Training crawlers decide whether future models learn from your content. Most B2B companies should allow both, verify access in server logs, and revisit the decision once a year. If you are unsure where you stand, the free AI visibility checker shows how engines currently see your brand, and what is llms.txt covers the optional companion file.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT search?
No. OpenAI treats GPTBot (training) and OAI-SearchBot (search) separately. OpenAI states that sites blocking OAI-SearchBot will not be shown in ChatGPT search answers, but blocking GPTBot alone affects training, not search inclusion.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode are part of Search and are controlled by Googlebot and standard snippet controls such as nosnippet.
Do AI crawlers respect robots.txt?
The major providers say their crawlers do. User-initiated fetchers such as ChatGPT-User and Perplexity-User are treated differently, and providers note robots.txt may not apply to requests a user triggers directly. Use firewall rules if you need hard enforcement.
Should I add llms.txt as well?
It is optional and separate from robots.txt. robots.txt controls access; llms.txt is a proposed file that summarizes your site for AI agents. See our page on llms.txt for details.
