ClaudeBot collects web content that may contribute to model training. Claude-SearchBot navigates the web to improve search result quality. Claude-User may access a site when a person asks Claude a question. Distinguish these names before you choose access rules or read Aiola Analytics.
Each bot serves a different purpose
Anthropic's crawler guidance separates model development, web search, and user-directed retrieval. Restricting ClaudeBot signals that future materials should be excluded from Anthropic's model training datasets. Disabling Claude-SearchBot prevents Anthropic from indexing that content for search. It may reduce your site's visibility and accuracy in search results. Disabling Claude-User prevents retrieval in response to a user query. It may reduce your site's visibility for user-directed web search.
You can allow search indexing and still block training. A ClaudeBot rule does not apply to Claude-SearchBot or Claude-User. Name every bot you want to restrict.
Match documented names in request headers
Anthropic's article names ClaudeBot, Claude-SearchBot, and Claude-User. It does not publish full User-Agent strings. Aiola's crawler pages mark the example strings as unverified. Match the token inside the full header. Do not assume a version suffix.
Aiola's directory says to match each token without case sensitivity. Search each documented name separately. A matching header can be spoofed. The header alone does not authenticate the sender.
Aiola also recognizes anthropic-ai. Anthropic's current article does not list that token. Do not treat it as one of the three documented bots. Check the header before you decide which rule applies.
Check the source against published addresses
Anthropic publishes one crawler IP list. The support article links that file for origin checks. Compare the source IP in trusted server or edge logs with the current prefixes. An address on the list means the crawler is coming from Anthropic.
The file is a single prefix list. It does not assign a range to each bot. Keep the User-Agent beside the address result so you can name the bot. Refetch the list regularly. A saved copy is not a permanent firewall list.
Give each bot its own access rule
Anthropic says its bots honor robots.txt. It also says they will not try to bypass CAPTCHAs. To block a bot, add a group in the top-level robots.txt file. Add that group on every subdomain you want to restrict.
Anthropic's published example is one group with Disallow: /. Use that same pattern for each bot you choose to block. Copy only the groups that match your policy. Put them in the existing file and leave unrelated rules in place.
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /The robots.txt guide pairs Allow: / with private-path exclusions and a Content-Signal line. A specific group needs its own private-path exclusions. Do not assume it inherits them from the wildcard group. Robots.txt asks cooperative crawlers which paths they may fetch. It does not protect confidential content. Keep private resources behind application access controls.
Blocking those published addresses may not reliably opt the site out. Anthropic says the block can impede its ability to read robots.txt. After deployment, open the served file on each hostname you changed. Check that response, not only the copy in your repository.
Anthropic supports the non-standard Crawl-delay extension in robots.txt. Its example pairs User-agent: ClaudeBot with Crawl-delay: 1.
Anthropic lists [email protected] for further questions or suspected malfunctions. Use an email that includes the domain you are asking about.
Send crawler hits through server-side tracking
Crawlers do not run JavaScript, so the browser tag cannot see them. One server or edge snippet from the setup guide posts each matching hit. Only known crawler user agents are stored.
Read visits in the AI view
The AI view shows crawl totals, bots, pages, and activity over time. It includes Answers, Indexing, Training and Other buckets, AI-answer referrals, crawler 404s, and discovery-file health.
Aiola names 70 crawlers. Aiola re-derives the bot, vendor, and kind from the User-Agent. The caller cannot choose that classification. ClaudeBot and anthropic-ai sit in Training. Claude-SearchBot sits in Indexing. Claude-User sits in Answers. The Analytics documentation describes these classifications.
A recorded crawl is not proof that the vendor cited the page. An empty period is not proof that the vendor cannot reach the site.
AI-answer referrals appear separately from the recorded crawler requests.
Try it in Aiola
Download Aiola free for macOS 12 or later and Windows 10 or later. Hosted Analytics needs Pro or the 7-day trial, and the pricing page lists both. Open the AI view to compare ClaudeBot, Claude-SearchBot and Claude-User.
