AI crawlers directory

ChatGPT-User

OpenAI

ChatGPT-User fetches a page after a person asks ChatGPT to visit or summarize it; it is not the automated training crawler.

Claude-User

Anthropic

Claude-User retrieves pages for user requests in Claude.

Perplexity-User

Perplexity

Perplexity-User visits a URL as part of a specific user action and may ignore robots.txt because the fetch is user initiated.

MistralAI-User

Mistral

MistralAI-User retrieves web pages for user questions in Vibe and can link to those sources.

Grok-DeepSearch

xAI

Grok-DeepSearch fetches sources while a Grok user runs DeepSearch.

Google-GeminiNotebook

Google

Google-GeminiNotebook retrieves sources for Gemini Notebook at a user’s request.

Google-Read-Aloud

Google

Google-Read-Aloud renders pages for user-requested text-to-speech and may serve cached results.

Google-Agent

Google

Google-Agent navigates and acts on the web at a user's request.

Copilot

Microsoft

Copilot fetches web content for a Microsoft Copilot interaction rather than performing Bing's general index crawl.

Amzn-User

Amazon

Amzn-User fetches current web information for user actions such as Alexa questions and may not follow every robots.txt directive.

meta-externalfetcher

Meta

meta-externalfetcher retrieves individual links at a user's request and supports agentic AI tasks.

DuckAssistBot

DuckDuckGo

DuckAssistBot crawls pages in real time for DuckDuckGo's cited AI-assisted answers; the content is not used to train AI models.

Kimi-User

Moonshot AI

Kimi-User handles user-initiated actions such as article summaries and live web retrieval.

Qwen-User

Alibaba

Qwen-User fetches web content in response to a Qwen user request.

OAI-SearchBot

OpenAI

OAI-SearchBot discovers and indexes pages so ChatGPT search can retrieve and cite current web results.

Claude-SearchBot

Anthropic

Claude-SearchBot crawls content to improve Claude search results.

PerplexityBot

Perplexity

PerplexityBot indexes pages for Perplexity search results and citations; Perplexity says it is not used for foundation-model training.

MistralAI-Index

Mistral

MistralAI-Index automatically indexes web content for Mistral search used by Vibe.

xAI-SearchBot

xAI

xAI-SearchBot discovers pages for xAI search retrieval used by Grok.

Google-CloudVertexBot

Google

Google-CloudVertexBot crawls and indexes website data for customer-created Agent Search data stores.

Googlebot

Google

Googlebot crawls and renders pages for Google Search indexing, including surfaces that supply search results to AI features.

Applebot

Apple

Applebot supplies Apple search experiences and can also collect data for foundation-model training and AI answer grounding.

Bingbot

Microsoft

Bingbot crawls and renders pages for the Bing index, which also supplies results to Microsoft Copilot.

Amzn-SearchBot

Amazon

Amzn-SearchBot crawls for Amazon search experiences and is not used for generative AI training.

TikTokSpider

ByteDance

TikTokSpider crawls pages for TikTok search, previews, and discovery surfaces.

Bytespider

ByteDance

Bytespider crawls for Toutiao Search.

meta-webindexer

Meta

meta-webindexer crawls content to improve Meta AI search results and enable citations and links in responses.

YouBot

You.com

YouBot indexes public pages for You.com's search engine and cited AI answers.

Kimi-SearchBot

Moonshot AI

Kimi-SearchBot analyzes page relevance and builds Kimi's search index.

Baiduspider

Baidu

Baiduspider discovers and indexes pages for Baidu Search.

Diffbot

Diffbot

Diffbot crawls pages for its search index and Knowledge Graph and excludes AI training.

ImagesiftBot

Hive

ImagesiftBot collects public images and page text into an index for similar-image search and retrieval.

msnbot

Microsoft

Microsoft retired MSNBot in favor of Bingbot starting October 1, 2010.

LinerBot

Liner

LinerBot builds Liner's search and reference index and follows its own robots.txt group.

QualifiedBot

Qualified

QualifiedBot crawls customer websites to populate the knowledge used by Qualified's AI SDR.

YandexBot

Yandex

Yandex's primary search crawler.

DuckDuckBot

DuckDuckGo

DuckDuckGo's search crawler.

SeznamBot

Seznam

SeznamBot retrieves and refreshes pages for Seznam's search engine.

PetalBot

Huawei

PetalBot is the crawler for Huawei's Petal Search.

Qwantbot

Qwant

Qwantbot builds and refreshes Qwant's search index and respects robots.txt.

GPTBot

OpenAI

GPTBot collects public web content that may be used to improve OpenAI's generative AI models.

ClaudeBot

Anthropic

ClaudeBot collects web content that may contribute to model training.

anthropic-ai

Anthropic

Anthropic's current crawler documentation lists ClaudeBot, Claude-User, and Claude-SearchBot.

GrokBot

xAI

GrokBot is an xAI crawler identity associated with Grok's automated web collection.

xAI-Web-Crawler

xAI

xAI-Web-Crawler is a broad xAI automated web-crawler identity.

xAI-Grok

xAI

xAI-Grok is an xAI crawler token associated with Grok content collection.

xAI-Bot

xAI

xAI-Bot is a general xAI automated crawler identity seen in server logs.

Grok

xAI

Grok is the broad Grok match token retained by Aiola for xAI crawler traffic not identified by a more specific token.

Google-Extended

Google

Google-Extended is a robots.txt product control for Gemini training and grounding, with no separate HTTP User-Agent.

Applebot-Extended

Apple

Applebot-Extended is a robots.txt data-use control that does not crawl webpages and has no separate HTTP User-Agent or IP feed.

Amazonbot

Amazon

Amazonbot collects content to improve Amazon products and services, and that content may train Amazon AI models.

meta-externalagent

Meta

meta-externalagent crawls content for foundation-model training and direct indexing to improve Meta products.

CCBot

Common Crawl

CCBot builds Common Crawl's publicly accessible web-crawl repository.

cohere-ai

Cohere

Cohere says it does not currently use bots to crawl web content for foundation-model training.

KimiBot

Moonshot AI

KimiBot collects content potentially used to train Kimi's foundation models.

QwenBot

Alibaba

QwenBot collects public pages for Qwen model and product development.

TongyiBot

Alibaba

TongyiBot is an Alibaba crawler identity associated with Tongyi model development.

AliyunBot

Alibaba

AliyunBot is an Alibaba Cloud crawler identity used for automated web collection.

ERNIEBot

Baidu

ERNIEBot collects public web content associated with Baidu's ERNIE model ecosystem.

YiyanBot

Baidu

YiyanBot is a Baidu crawler identity associated with the ERNIE/Yiyan generative AI service.

ChatGLM-Spider

Zhipu AI

ChatGLM-Spider collects public web content for Zhipu AI's ChatGLM model ecosystem.

DeepSeekBot

DeepSeek

DeepSeekBot is an automated crawler associated with DeepSeek model and product development.

AI2Bot

Allen AI

AI2Bot finds web content used to train open language models.

Timpibot

Timpi

Timpibot crawls public pages for Timpi's decentralized search index.

Doubaobot

ByteDance

Crawler for ByteDance's Doubao assistant.

Google-InspectionTool

Google

Google-InspectionTool fetches pages for Search Console URL Inspection and the Rich Results Test.

GoogleOther

Google

GoogleOther is Google's general-purpose crawler for product teams, including one-off internal research and development.

OAI-AdsBot

OpenAI

OAI-AdsBot visits landing pages submitted as ChatGPT ads to check safety and determine ad relevance.

meta-externalads

Meta

meta-externalads crawls content to improve Meta advertising and other business products and services.

facebookexternalhit

Meta

facebookexternalhit gathers shared-page metadata for link previews on Facebook, Instagram, and Messenger.