Google-Extended controls Gemini training and grounding, and it does not control Search indexing. Google's crawler documentation describes a robots.txt token instead.
It covers Gemini training and grounding
The common crawlers page says the token manages training for future Gemini models. Those models power Gemini Apps and the Vertex AI API for Gemini. A disallow excludes matching paths from these permitted uses. The Google-Extended page files the token under Training.
Grounding supplies indexed content when a model answers a prompt. The named surfaces are Gemini Apps and Grounding with Google Search on Vertex AI.
It is not a ranking signal
Google states that the token is not a ranking signal in Google Search. Googlebot rules govern Search crawling, including Discover and Search features.
AI Overviews use another control
AI Overviews are a Search feature, so this token does not set appearance there. Google's AI features guide names Googlebot robots.txt as the crawl control for Search. A supporting link needs an indexed page that is eligible to show a snippet. Search Console's generative AI control, under Settings, covers AI Overviews, AI Mode, and generative AI features in Discover. Excluding a site there stops impressions and traffic from those features.
To limit text shown from a page, Google lists four preview controls. Google names nosnippet, data-nosnippet, max-snippet, and noindex for those limits. Those controls change Search display, and they are separate from this token. The preview control must be visible to Googlebot or it will not apply. Google may take days or months to recrawl and apply the change.
That Search Console control does not affect AI training. Google describes a delay of a few days, with caching sometimes extending exclusion beyond one or two days. Google's help page points training limits back to Google-Extended. The control reached all websites on August 31, 2026.
Use this robots.txt group
An example group is published on the common crawlers page. The token on the user-agent line is Google-Extended. One archive path is allowed, and the parent archive is disallowed. For a whole-site restriction, remove the allow line and set disallow: /. Leave every other user-agent group in place when you add this one. Google's robots.txt specification explains how path rules and exceptions interact.
user-agent: Google-Extended
allow: /archive/1Q84
disallow: /archive/Why logs skip that name
Google-Extended has no separate HTTP request user agent. Existing Google request identities carry the traffic, so a log search cannot isolate the token's permitted uses.
Aiola's Googlebot page provides a sample request string for the Search crawler. Its directory record classifies Googlebot as Indexing and links Google's published common crawler IP ranges. These records describe crawling and retrieval, rather than evidence of a particular page's use in Gemini training.
A request claiming to be Googlebot still needs verification. Google's verification guide provides DNS and IP checks. For a manual check, take the requesting IP address from your server logs. Run a reverse DNS lookup and inspect the returned hostname. For common crawlers, Google lists googlebot.com and geo.googlebot.com hostname patterns.
Then run a forward DNS lookup on that hostname. Confirm that it resolves back to the original requesting IP address. Alternatively, match the address against Google's published common crawler IP ranges. A matching user agent alone does not establish that Google sent the request.
Vertex AI crawls use another token
Google-CloudVertexBot is a different crawler with its own request substring. Its directory record describes indexing website data for customer-created Agent Search data stores. Aiola classifies that crawler as Indexing. That indexing purpose does not establish foundation-model training. Google documents separate crawling preferences for this crawler, without effects on Google Search or other products.
Watch Googlebot in Analytics
AI crawler analytics shows recorded Search crawler activity. Analytics classifies Googlebot as Indexing. Google-Extended has no separate HTTP request user agent.
The AI view shows crawl totals, bots, pages, and activity over time. It groups hits into Answers, Indexing, Training, and Other. AI-answer referrals, crawler 404s, and discovery-file health also appear there.
Crawlers do not run JavaScript, so the browser tag cannot see them. One server or edge snippet from the setup guide posts each matching hit. Only known crawler user agents are stored.
A recorded crawl is not proof that the vendor cited the page. An empty period is not proof that the vendor cannot reach the site.
Try it in Aiola
Download Aiola free for macOS 12 or later and Windows 10 or later. Hosted Analytics needs Pro or the 7-day trial, and the pricing page lists both. Open the AI view and review Googlebot activity in Indexing.
