OAI-SearchBot

What is OAI-SearchBot?

OAI-SearchBot is the web crawler OpenAI uses to find and index pages for ChatGPT’s search features. When ChatGPT searches the web and shows sources, the pages it can cite come largely from what OAI-SearchBot has crawled. Site owners control it with the OAI-SearchBot token in robots.txt, separately from OpenAI’s training crawler, GPTBot.

OpenAI introduced the crawler in 2024 alongside its SearchGPT prototype, which later became ChatGPT search. Its existence matters to marketers because it splits a decision that used to be bundled: whether an AI company may train on your content, and whether its search product may show and link to it.

How it works

OpenAI documents several agents, each with its own purpose:

Agent	Purpose	robots.txt
OAI-SearchBot	Surfaces sites in ChatGPT search	Respected
GPTBot	Collects content that may train models	Respected
ChatGPT-User	Fetches pages for a user’s request	May not apply

Details from OpenAI’s bot documentation:

• The user agent string contains OAI-SearchBot/1.x plus a link to openai.com/searchbot. OpenAI notes version numbers may change, so match on the token, not the full string.
• OpenAI publishes its IP ranges at openai.com/searchbot.json, which helps you verify genuine requests.
• Sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers” but “can still appear as navigational links.”
• Changes take roughly 24 hours to apply after a robots.txt update.
• If both OAI-SearchBot and GPTBot are allowed, OpenAI may use one crawl for both purposes.

From SEO to AEO and GEO

The classic SEO equivalent is Googlebot or Bingbot: the crawler that decides whether you can be found at all. OAI-SearchBot plays that role for ChatGPT search. What changed is the output. Instead of ten ranked links, ChatGPT composes an answer and cites a handful of sources, so being crawlable is necessary but not sufficient; the page also has to contain a passage worth quoting.

What carries over is nearly all of technical SEO: clean server responses, fast pages, content present in the initial HTML, sensible internal links and no accidental blocks. Many AI crawlers fetch raw HTML and may not run client-side JavaScript fully, so server rendering matters at least as much as it did for search engines.

What to do about it

1. Check robots.txt for any OAI-SearchBot group or a wildcard rule that catches it.
2. Check the edge. CDN bot protection, WAF challenges and rate limits often block AI crawlers even when robots.txt allows them. Look for 403 and 429 responses in logs.
3. Verify in logs that requests with the OAI-SearchBot token come from OpenAI’s published IP ranges and receive 200 responses.
4. Separate the policies. Many B2B teams allow OAI-SearchBot everywhere public and decide on GPTBot separately.
5. Test outcomes. After a change, wait a day or more, then run your key buyer prompts in ChatGPT search and see whether your pages appear as sources.

Common mistakes

• Blocking it with a “block all AI” template and then wondering why ChatGPT never cites the site.
• Treating GPTBot and OAI-SearchBot as one bot. They have different purposes and different consequences.
• Trusting the user agent alone. Scrapers spoof it; verify against OpenAI’s IP list.
• Expecting instant results. Crawling, indexing and citation all take time.

Related terms

• GPTBot
• AI Crawler
• ChatGPT Search
• AI Referral Traffic
• Log File Analysis
• How AI engines cite sources

Sources

• Overview of OpenAI crawlers, OpenAI developer docs
• Robots.txt introduction, Google Search Central (syntax reference)