GPTBot is OpenAI's web crawler that collects publicly available pages that may be used to train its AI models. Site owners can allow or block it with robots.txt.
What is GPTBot?
GPTBot is the web crawler OpenAI uses to gather publicly available content that may be used to train its models. It identifies itself with the user-agent token GPTBot, and site owners can control its access through robots.txt. OpenAI announced it in August 2023 and lists it alongside its other crawlers in its bot documentation.
How it works
OpenAI operates more than one crawler, each with a different purpose:
| User agent | Purpose |
|---|---|
| GPTBot | Crawls content that may be used for model training |
| OAI-SearchBot | Finds and indexes sites for ChatGPT search results |
| ChatGPT-User | Fetches a page when a user’s request requires it |
Because these are separate, a site can, for example, block GPTBot for training while still allowing OAI-SearchBot so pages can appear and be cited in ChatGPT search.
A robots.txt rule to block GPTBot looks like this:
User-agent: GPTBot
Disallow: /Why it matters for AI search
Training data shapes how models describe your brand when they answer without searching. Blocking GPTBot on a marketing site removes your own pages from future training sets, which can leave models relying more on third-party descriptions. Many companies also block AI crawlers by accident through a CDN or firewall bot setting.
Examples
- A B2B SaaS company allows GPTBot on its marketing site and docs but disallows its customer portal paths.
- A publisher blocks GPTBot for training but allows OAI-SearchBot to keep ChatGPT search referral traffic.
- An SEO audit finds a CDN rule returning errors to all AI crawlers, including GPTBot.
Other AI companies follow the same pattern. Anthropic, Perplexity and others publish their own crawler names and documentation, so a robots.txt review should cover each engine your buyers use, not only OpenAI.
How to apply it
- Check robots.txt for GPTBot, OAI-SearchBot and ChatGPT-User rules.
- Decide on training access and search access separately.
- Check server logs to confirm the crawlers can actually fetch pages.
- Review CDN and WAF bot settings that may block AI user agents.
Related terms
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT search?
Not necessarily. OpenAI documents a separate crawler, OAI-SearchBot, for surfacing sites in ChatGPT search results. Blocking GPTBot affects training use; blocking OAI-SearchBot affects search visibility.
How do I block GPTBot?
Add 'User-agent: GPTBot' followed by 'Disallow: /' to robots.txt, or disallow specific paths only.
Should B2B companies block GPTBot?
Most B2B marketing sites allow it for public pages, because being part of training data helps models describe the brand. Gated, private or licensed content is a different decision.
Guides that cover GPTBot
-
Answer Engine OptimizationBeyond Keywords: Why AEO is Replacing Traditional SEO for B2B SaaS
Traditional SEO is dying. Learn how Answer Engine Optimization (AEO) is replacing standard search and why B2B SaaS brands must focus on entity authority now.
-
AEOHow to Get Cited by ChatGPT: Reverse-Engineering AI Answer Sources for B2B Topics
Learn how to reverse-engineer ChatGPT citations for B2B SaaS. Master AEO and GEO strategies to get your brand cited in AI-generated answers.
-
Generative Engine OptimizationInsights on Generative Search Engine Optimization
Explore the latest insights on Generative Search Engine Optimization (GEO), including strategies, benefits, and its future impact on digital marketing. Learn how to optimize for AI-driven search and stay ahead.
- robots.txt
Robots.txt: The Technical SEO Guide to Crawler Management
How robots.txt works, the syntax for User-agent, Disallow, Allow and Sitemap directives, how to create and test the file, and advanced crawler rules.
