AI crawlers can request pages without running your analytics script. Browser analytics misses requests that never execute its tag. You can read server logs, a CDN bot report, or named records in Aiola Analytics.
What do server logs show about bots?
Configured access logs can record requests reaching your server. Include the path, status code, time, and user agent in your log format. Grep that user agent when you want one bot. A match shows the URL it requested and the status you returned.
Confirm the log format actually prints the user agent. Some formats keep only the path, the status, and the time. With no agent field, the search cannot name the bot. Keep logs covering your chosen date range before rotation removes them.
ChatGPT-User handles certain user-triggered requests in ChatGPT. OAI-SearchBot supports ChatGPT search results. GPTBot crawls content that may be used for model training, according to OpenAI’s crawler documentation. The ChatGPT-User page and the crawler directory list the strings to search.
Count matching lines within the path and date range you need. A user agent identifies a declared bot, rather than proving how fetched content was used.
What does a CDN bot report add?
A CDN report can group requests by its own bot classifications. For example, Cloudflare Bot Analytics shows traffic types, detection sources, and request attributes. Availability and detail depend on your Cloudflare plan.
Cloudflare reports sampled data, with the sample rate shown in tooltips. Compare the same time window when checking its report against server logs.
Edge caching and the place you instrument change what you can see. A cached response may never appear in the origin access log. Instrumentation inside the app misses any request that never reaches it. Check one known fetch in each layer before treating the bot as absent.
Why do page scripts miss these bots?
Crawlers do not run JavaScript, so the browser tag cannot see them. One server or edge snippet from the setup guide posts each matching hit. Only known crawler user agents are stored.
Which crawlers does Aiola name?
The AI view shows crawl totals, bots, pages, and activity over time. It also shows AI-answer referrals, crawler 404s, and discovery-file health.
Fetch hits land in Answers, while search and classic index hits land in Indexing. General crawling goes into Training. Ads, social, and remaining kinds go into Other.
A recorded crawl is not proof that the vendor cited the page. An empty period is not proof that the vendor cannot reach the site.
How do the three methods compare?
Aiola Analytics names 70 crawlers. Compare where each method records requests and which traffic it can miss.
| Method | Where it is recorded | How the bot is named | Main limit |
|---|---|---|---|
| Server logs | Origin access log | You match the user agent | Only strings you choose to search |
| CDN bot report | At the edge, before your app | The CDN's own classes | Broad classes need a user agent check |
| Aiola Analytics | A beacon from your server or edge | Known user agent, then a bucket | Unknown agents and disallowed hosts are dropped |
Which method should you use?
After you deploy the server or edge snippet, request a page with a matching crawler user agent. Then open the AI view and look for that path. This checks whether the request reaches your integration and appears in the selected period. Compare that path with the request in your server logs.
Grep the logs when you need one path and one status today. Open the CDN report when you want volume the edge already counted. Send the server beacon when named crawlers should sit beside human visits.
A person who clicks a ChatGPT link is not a crawler line. That visit can show an assistant referrer when the browser sends one. The AI view lists those referrals apart from crawl totals. Count that person with the browser tag, not with the bot rows.
A successful assistant fetch does not prove a recommendation followed. Use crawler rows to inspect requested URLs, and browser events to examine subsequent human traffic.
Try it in Aiola
Download Aiola free for macOS 12 or later and Windows 10 or later. Hosted Analytics needs Pro or the 7-day trial, and the pricing page lists both. Open the AI view to compare crawler requests and assistant referrals.
