Is Google's common crawler list complete?

Google says the common crawler list is not exhaustive. It covers requestors likely to show up in logs and that Google has been asked about.

Do user-triggered fetchers obey robots.txt?

User-triggered fetchers ignore robots.txt because a person requested the fetch. Common crawlers always respect robots.txt rules on automatic crawls.

Does Google-InspectionTool use the same ranges?

Google lists Google-InspectionTool with the common crawlers in common-crawlers.json. Rules for that token affect the Rich Result Test and URL inspection, and they do not change Google Search.

Will a robots.txt block keep a URL out of Google Search?

A robots.txt disallow can stop the crawl and still leave the URL in search results. Use noindex to prevent indexing, while allowing Google to crawl the page and read that rule.

How much of a page does Googlebot fetch?

For Google Search, Googlebot fetches the first 2MB of a file, or the first 64MB of a PDF. The size limit applies to uncompressed data, and other Google crawlers may differ.

Verify Googlebot by IP Ranges

Match the request address to Google's published ranges, or confirm it with reverse and forward DNS. Google warns that a googlebot user agent is often spoofed, so the header is not proof. Both checks are documented on Google's verification page.

What do you copy from the log?

Pull the connecting address from your server or edge log. Keep the full user-agent header, path, and time with that address. Check this same address with DNS or against the published ranges.

How do you check DNS both ways?

Google's page splits the work into manual and automatic checks. Manual host lookups are for one-off addresses and cover most cases. Automatic matching against the JSON files is for large scale lookups.

Google's manual method uses the host command on the logged address. The returned hostname must belong to googlebot.com, google.com, or googleusercontent.com. Then resolve that name forward and require the original address in the answer. Any other name, or a different forward answer, fails the check.

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

Google's first example returns crawl-66-249-66-1.googlebot.com for 66.249.66.1. The forward lookup returns the original address. Common crawler hostnames look like crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Another documented address, 35.247.243.240, reverses to the name geo-crawl-35-247-243-240.geo.googlebot.com.

The third documented example starts from the address 66.249.90.77. A reverse lookup on it returns the name rate-limited-proxy-66-249-90-77.google.com. Forward lookup of that hostname returns the same address, 66.249.90.77. This hostname pattern is the special-case crawler mask from the verification page.

Which files publish the ranges?

For many requests, match the address to Google's JSON range files instead of DNS. Google publishes common crawlers such as Googlebot in common-crawlers.json. Special-case crawlers such as AdsBot are published in special-crawlers.json. User-triggered fetchers are split across user-triggered-fetchers.json, user-triggered-fetchers-google.json, and user-triggered-agents.json. The older Search API URL for googlebot.json redirects to common-crawlers.json.

The common crawler file includes creationTime and a prefixes list containing IPv4 and IPv6 CIDR ranges. One IPv4 entry was 66.249.66.0/27, which contains the sample address 66.249.66.1. Another entry, 35.247.243.240/28, covers the geo-crawl sample 35.247.243.240.

Test the logged address against each prefix as a network, not as a text search. An address can fall inside a prefix without matching its first number. Ranges change, so fetch the current file when you check a new request. Addresses for products such as Apps Script belong on the separate Google IP list.

Google documents user-triggered hostnames as ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com. Fetchers that Google controls come from user-triggered-fetchers-google.json and use a google.com hostname. Rows in user-triggered-fetchers.json resolve to gae.googleusercontent.com hostnames instead.

Which strings does Googlebot send?

On the Googlebot page, Googlebot is the generic name for two Search crawlers. Smartphone simulates a person on a mobile device. Desktop simulates a person on a desktop computer. You identify the subtype from the HTTP user-agent header. Most Search crawl requests use the mobile crawler, and both share the robots.txt token Googlebot.

Google publishes these request strings for Googlebot on its common crawler list. The Chrome/W.X.Y.Z portion is a placeholder for the current Chromium version, not a literal search key. Preferences for that token affect Google Search, Discover, Images, Video mode, and Google News. Two shorter strings appear rarely, so match them along with the long forms.

Smartphone: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Desktop: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Rare: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Rare: Googlebot/2.1 (+http://www.google.com/bot.html)

Googlebot Image sends the string Googlebot-Image/1.0, and it also matches the Googlebot robots token. Video fetches use Googlebot-Video/1.0 and also match that same token. News crawling has no request string of its own and uses the Googlebot strings above. Its robots.txt token is Googlebot-News, which affects the Google News product.

How do you separate GoogleOther?

GoogleOther is a common crawler with its own header and the robots.txt token GoogleOther. Google describes it as a generic crawler for product teams, including one-off internal research. Its preferences do not target a specific product. These two strings are the ones Google publishes.

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36

GoogleOther is published on the same common-crawlers.json list as Googlebot. A matching range or a passing DNS check does not name the token. Read the full header once the address check passes. The Googlebot and GoogleOther pages keep those identities separate.

Where does Aiola list each bot?

The analytics docs describe the AI view. It shows crawl totals, bots, pages, activity over time, AI-answer referrals, crawler 404s, and discovery-file health. Its buckets are Answers, Indexing, Training, and Other. Googlebot maps to Indexing, and GoogleOther maps to Other.

Crawlers do not run JavaScript, so the browser tag cannot see them. One server or edge snippet from the setup guide posts each matching hit. Only known crawler user agents are stored. Use your server log for the source address needed by the range or DNS check.

A recorded crawl is not proof that the vendor cited the page. An empty period is not proof that the vendor cannot reach the site.

Try it in Aiola

Download Aiola free for macOS 12 or later and Windows 10 or later. Hosted Analytics needs Pro or the 7-day trial, and the pricing page lists both. Open Analytics in your project to review Googlebot and GoogleOther separately.

Verify Googlebot by IP: Google's Published Ranges