Diffbot · AI answers
Diffbot-User
Identify Diffbot-User by its user agent. Check request identity, review robots.txt controls and track visits in Aiola Analytics.
Key facts
Diffbot-UserWhat Diffbot-User is
Diffbot fetches on behalf of a human user. Only the token is documented; no full User-Agent example or IP list is published.
In Aiola's taxonomy, AI answers means fetched to answer a user's question now.
Diffbot-User user agent
Look for the match token inside the complete HTTP User-Agent header. Tokens can be spoofed, so use the network checks below for authentication.
Diffbot-UserDiffbot-UserOfficial IP ranges
Diffbot has not published official IP ranges that Aiola successfully fetched for Diffbot-User. Aiola does not list cloud-provider ranges or community guesses as if they authenticated this crawler.
Verify a Diffbot-User request
No IP authentication method is documented in this record for Diffbot-User. Its user-agent token is a claim, not proof of identity.
Use the exact header token for classification and apply your own access controls. Check the vendor documentation for any newly published IP, token or header verification method.
Control Diffbot-User with robots.txt
Diffbot-User is user-triggered. Diffbot's controls may differ from automated crawling, so do not assume a robots.txt block is an authentication boundary.
User-agent: Diffbot-User
Allow: /User-agent: Diffbot-User
Disallow: /Aiola Analytics
See every Diffbot-User visit on your site
Aiola tracks 80 AI, search and link-preview crawlers next to your human traffic, revenue and errors. See when Diffbot-User arrives, which pages it requests and which requests fail. Then ask a coding agent in the same app to fix those pages. Track Diffbot-User visits with AI crawler analytics.
Common questions
Everything your agents need, in one app
Errors, user feedback, GitHub and analytics live next to your coding agents. Every issue becomes a task an agent can take.