Detect GPTBot, ClaudeBot and PerplexityBot by reading the request's full user agent at your server or edge. Verify the source IP against the matching vendor feed before trusting that identity. Then choose access rules for training, search and user requests separately.
How do you detect which crawler requested a page?
Start with one request, rather than a total labeled “AI traffic.” Keep its timestamp, hostname, path, response status, full user agent and source IP together.
A user-agent string is a claim that another client can copy. Matching a bot name identifies a candidate; checking its network address adds the vendor's documented origin check.
Look for these identities before choosing a feed:
| Claimed identity | Purpose | Reference |
|---|---|---|
| GPTBot | Content collection for possible model training | GPTBot user agent and IP ranges |
| ClaudeBot | Content collection for possible model training | ClaudeBot user agent and IP ranges |
| PerplexityBot | Search discovery and links | PerplexityBot user agent and IP ranges |
OpenAI's example contains GPTBot/1.4, and Perplexity's contains PerplexityBot/1.0. Both vendors say versions may change in their OpenAI and Perplexity crawler documentation.
Use the documented token instead of requiring one fixed browser wrapper or version. Anthropic documents ClaudeBot, but Aiola's directory marks its full sample string as unverified.
Keep the original header when creating a shorter bot label. That lets you inspect future changes without losing the evidence behind an earlier classification.
How do you verify crawler IPs against official feeds?
Select the feed for the claimed identity, then test whether the source address belongs to a published CIDR range. Searching for an exact address string can miss a valid address inside a larger range.
| Request claims | Official feed |
|---|---|
| GPTBot | OpenAI GPTBot JSON |
| ClaudeBot, Claude-SearchBot or Claude-User | Anthropic bots JSON |
| PerplexityBot | PerplexityBot JSON |
These feeds expose a prefixes list containing ipv4Prefix entries. Read the current feed instead of copying addresses from an old article or screenshot.
Follow this sequence for each request:
- Take the source address from the server or edge record for that request.
- Fetch the official feed matching its claimed bot identity.
- Parse each CIDR with an IP address library and test network membership.
- Save the matching prefix, feed URL, feed timestamp and time of your check.
- Keep unmatched requests separate for investigation before changing access rules.
Anthropic's crawler guidance says an address on its published list indicates an Anthropic crawler. The shared list cannot distinguish ClaudeBot from Claude-User; retain the user agent alongside the match.
Likewise, do not assume different OpenAI feeds have no overlapping prefixes. A matching range supports the origin check without identifying the request's purpose by itself.
If nothing matches, check the address you captured and whether your saved feed is current. Record “unverified” rather than turning a failed comparison into a confirmed vendor identity.
Refresh stored ranges when the official feeds change. Perplexity explicitly recommends combining the user agent and the appropriate IP list in firewall rules.
These sources publish IP feeds for the bots covered here. This workflow uses those feeds rather than assuming a reverse-DNS suffix establishes identity.
How should you handle search bots and user fetchers separately?
GPTBot is not OAI-SearchBot, and ClaudeBot is not Claude-SearchBot. Keep their search identities separate when reading requests or writing access rules.
Use OAI-SearchBot user agent and IP ranges with OpenAI's search feed. Use Claude-SearchBot user agent and IP ranges with Anthropic's shared feed.
User fetchers need their own checks:
- ChatGPT-User user agent and IP ranges links to its separate OpenAI feed.
- Claude-User user agent and IP ranges uses Anthropic's shared bot feed.
- Perplexity-User user agent and IP ranges uses its own Perplexity feed.
OpenAI says robots.txt may not apply to ChatGPT-User requests because a user initiated them. Perplexity says its user fetcher generally ignores robots.txt for that reason. Anthropic says its documented bots honor robots.txt, including Claude-User.
For the OpenAI distinction, see OAI-SearchBot vs GPTBot vs ChatGPT-User.
If your policy allows public search but excludes training collection, merge these groups into your existing robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Perplexity describes PerplexityBot as a search bot, rather than a foundation-model training crawler. It belongs with search access in this example.
Add your own excluded paths to each allowed group. Specific groups do not inherit a wildcard group's exclusions, as the AI bots robots.txt guide explains.
Preserve existing sitemap lines and unrelated groups. Read the served /robots.txt after deployment, and check each hostname separately. Keep private content behind application access controls; robots.txt expresses crawl preferences.
How does Aiola record crawler requests and help agents fix failures?
Aiola crawler analytics records recognized crawler requests through a server or edge snippet. Many crawlers skip browser JavaScript, so the browser tag alone misses those requests.
Choose the implementation for your stack in the Analytics setup guide. It sends matching requests with the public tracking ID, hostname, path and user agent. Country and response status are optional fields.
Send the actual status where your capture point can observe it. An omitted status defaults to 200, so a recorded success can reflect missing status data.
The snippet recognizes user agents; that classification is separate from your source-IP verification. Keep the address check in your server or edge investigation.
Open Analytics, then AI, to inspect bots, pages, crawl activity and crawler 404s. Ask your coding agent to investigate affected URLs, check redirects and propose a fix. Review the diff and relevant checks before publishing, then inspect fresh requests.
A crawl record does not prove that an assistant cited your page. An empty period means no matching event arrived, rather than proving the vendor cannot reach your site.
Try it in Aiola
Your coding agent can read crawler evidence alongside the repository and turn a failed page into a proposed fix. Aiola brings building, running and supporting products into one app; hosted Analytics is part of Pro. Download Aiola to keep production evidence within your agent's reach.
