Google-Extended

What is Google-Extended?

Google-Extended is a product token used in robots.txt that lets site owners control whether content Google crawls from their site may be used to train future Gemini models and to ground answers in Gemini Apps and Grounding with Google Search on Vertex AI. It is a usage control, not a crawler, and does not affect Google Search.

Google announced the token on September 28, 2023, in a post titled “An update on web publisher controls”. It arrived shortly after OpenAI published GPTBot, when publishers were asking every AI company for a way to opt out of model training. Google’s answer was to keep one crawling infrastructure and add a separate permission flag on top of it.

How it works

You add a group to robots.txt that names the token:

User-agent: Google-Extended
Disallow: /

Key mechanics, per Google’s crawler documentation:

• No separate user agent. Google-Extended “doesn’t have a separate HTTP request user agent string.” Pages are fetched by Google’s existing crawlers; the token only governs downstream use.
• Training and grounding. It covers training of Gemini models and grounding, where Search index content is supplied to a model at prompt time in Gemini Apps and Vertex AI.
• No Search impact. Google states it does not affect a site’s inclusion in Google Search and is not used as a ranking signal.
• Path-level control. Like any robots.txt group, you can disallow only certain directories.

From SEO to AEO and GEO

In classic SEO there was one Google decision to make: let Googlebot crawl or not. Generative AI split that into several decisions, and Google-Extended is one of them.

Question	Classic SEO control	AI-era control
Can Google index and rank this page?	Googlebot rules, noindex	Unchanged
Can AI Overviews and AI Mode show it?	Not applicable	Googlebot rules plus nosnippet, max-snippet
Can Gemini train on or ground with it?	Not applicable	Google-Extended

What carries over: robots.txt syntax, path-level rules and the need to test changes. What changed: a single “allow Google” policy now has separate training, grounding and Search dimensions, and the controls do not map one-to-one onto Google’s product names. Many SEO teams assume Google-Extended controls AI Overviews. Google’s own AI features page does not say that, and it routes AI Overviews controls through standard Search directives instead.

What to do about it

1. Audit robots.txt for an existing Google-Extended group. Many CMS plugins and “block AI” templates add it silently.
2. Decide by content type. Product pages, docs, pricing and comparisons usually benefit from being available to Gemini. Gated reports or licensed data may warrant a disallow.
3. Use snippet controls for Search surfaces. If a page must not be quoted in AI Overviews, use nosnippet or data-nosnippet on the specific passage, and accept that this also limits classic snippets.
4. Write the policy down alongside your rules for GPTBot, ClaudeBot and other AI crawlers so the whole team applies one consistent stance.
5. Re-check quarterly. Google’s documentation on AI controls has changed since 2023 and may change again.

Common mistakes

• Expecting log entries. There is no Google-Extended bot to see in server logs.
• Blocking Googlebot to stop AI use. That removes the site from Search entirely, which is rarely the goal.
• Assuming it controls AI Overviews. Google does not document it that way.
• One rule for the whole site. Path-level rules let you protect paid content while keeping marketing pages open.

Related terms

• AI Crawler
• GPTBot
• ClaudeBot
• Robots File
• AI Overviews
• Grounding
• Google AI Overviews optimization

Sources

• Google’s common crawlers (Google-Extended), Google Search Central
• AI features and your website, Google Search Central
• An update on web publisher controls, Google, September 2023