Track AI crawler visits and assistant referrals

A crawler request tells you that a matching user agent requested a page. A human referral tells you that someone arrived with an assistant referrer. A citation is an answer that actually names or links to your content. Combining the first two signals helps you understand discovery and acquisition, but neither proves the third.

Keep those observations separate in your reporting. More training crawls do not mean more customers. A successful assistant fetch does not establish that a recommendation followed. Aiola Analytics brings crawler activity and browser traffic into one project. Investigate their relationship while keeping them as separate metrics.

Many crawlers do not execute your browser JavaScript. A browser tag alone therefore cannot provide a complete view of bot requests. Send recognized crawler requests from your server or edge to POST https://aiola.app/api/bot-crawl-ingest. Keep the browser tracker for visitors.

The documented crawler payload contains tracking_id, url_path, url_host, and ua, with optional country and status. Use the public Analytics tracking ID and a hostname allowed by the site configuration. This endpoint does not use the secret project API key used by App Logs and Feedback. Unknown crawler user agents and hosts outside the configured domain are not stored.

Send the response status actually observed at the capture point. Ingest defaults to status 200 when the field is omitted. A recorded success may reflect missing data rather than a verified response. Edge caching and the location of your instrumentation also determine which requests you observe. Check a known request against your server logs before using the dashboard as your baseline.

Aiola groups fetch activity into Answers and search or classic indexing into Indexing. General crawling goes into Training; ads, social and remaining kinds go into Other. The categories describe recognized request types rather than a vendor’s endorsement of your site.

ChatGPT-User represents an on-demand OpenAI assistant fetch. OAI-SearchBot is associated with search indexing; GPTBot with corpus or training crawling. Those names belong in different buckets even though they come from one organization. The crawler directory explains identities and available verification information.

Classification uses the request User-Agent and can be spoofed. Where a vendor publishes IP ranges or reverse-DNS guidance, use those checks to investigate identity. A missing row means Aiola did not receive a matching beacon in the selected period. It does not prove a vendor is blocked, never accessed the site, or cannot discover the content elsewhere.

The Analytics AI view includes crawl totals, bots, pages, time trends and purpose buckets. It also includes AI-answer referrals, uncrawled popular pages, crawler 404s and discovery-file health. Start at the page level. Repeated crawler hits on an old URL may need a redirect; an important new page may need internal links.

Compare your capture logs with the recorded error before changing routing. Fix the underlying missing page or stale link, then observe subsequent requests. Requests to robots.txt, llms.txt and a sitemap show discovery-file activity. They do not establish whether an assistant used those files in an answer.

Aiola computes a deterministic 0–100 visibility score. It combines popular-page crawl coverage, crawler breadth, AI-referral visitors, assistant breadth and discovery-file health. It is available through the authenticated overview and MCP reporting path. The current AI dashboard does not show a numeric score card. Use its components as operational clues rather than a ranking or citation guarantee.

Install the browser script with your tracking ID and allowed domain to capture visits and events. Then compare assistant referrers with landing pages and the events that matter to your application. A docs visit followed by a signup is a more useful acquisition question than simply counting bot hits.

Goals use tracked event names, while saved funnels use page and event definitions. Goals are browser-local on the device; saved funnels are project definitions. Session journeys let you inspect the recorded sequence behind a conversion. Missing referrers, blocked scripts, and incomplete instrumentation can all reduce what you observe.

For Stripe attribution, Checkout Sessions must carry aiola_visitor_id and aiola_session_id metadata. Manual revenue can be recorded with the documented __revenue event. Revenue association depends on those identifiers and the collected journey. It does not establish that a crawler request caused a purchase. Keep bot volume, visitor conversions, and attributed revenue as separate measures.

Create the site with its production domain, install the tag, and configure server-side crawler capture. Use Setup Health to investigate a missing tag or tracking-ID mismatch. Proxying script delivery can bypass blockers. Proxying event collection loses trustworthy visitor geography because the gateway sees the proxy connection.

Hosted Analytics and Setup Health need Pro or an active trial. Monthly and yearly Pro tiers are based on analytics volume. Billable events are pageviews and custom events, excluding the documented leave, identify, and scroll events. Do not use crawler totals as a substitute for the Usage page’s event allowance.

The eligible trial lasts seven days for first-time customers on Pro Monthly with 100k events and requires a card. GUI Lifetime does not include hosted Analytics. Without Pro or an active trial, browser and bot collectors reject events rather than storing them. Monthly quota overages behave differently: collection continues while dashboard access locks until upgrade or reset.

Use Analyze with AI to open a scoped reporting thread, or query the same project through hosted MCP. Ask for the affected URLs, time range, crawler categories, and referral evidence behind each suggestion. A useful next task might be repairing a broken reference page, improving internal linking, or checking a signup funnel.

Aiola desktop 2.0.8 supports macOS 12+ and Windows 10+ 64-bit. Download is free; provider access is separate and Aiola adds no token markup. Keep a recorded sample of actual assistant answers if your goal is citation measurement. Aiola’s crawler and referral data can guide that investigation, but do not replace it.