AI2Bot, MistralAI-User, DeepSeekBot and QwantBot have different purposes in Aiola's crawler directory. AI2Bot gathers training content, MistralAI-User handles user requests, and QwantBot builds a search index. Aiola classifies DeepSeekBot as Training, but its record has no vendor documentation link.
Who runs each crawler
Ai2 publishes the AI2 Bot on its crawling notice. The page says the bot explores certain domains and finds web content for open language models. Questions and concerns about the crawler can be sent to Ai2, as the notice asks. Aiola's crawler directory lists the vendor as Allen AI.
Mistral documents MistralAI-User on its robots page. The token is for user actions in Vibe. When a user asks Vibe a question, it may visit a page and add a source link. Mistral says this token does not crawl automatically and does not collect training content.
Aiola's DeepSeekBot record has no vendor documentation link. Aiola's record calls it an automated crawler tied to model and product development. The directory classifies that work as training because the fetch is not started by a user. Its last verification date is July 11, 2026.
Qwant's help page calls QwantBot the official indexing robot. It crawls the web to build and refresh the search index. Aiola's crawler directory records its vendor as Qwant.
Which string appears in the log
Look for the token inside the full User-Agent header. The documented tracking snippets match crawler tokens without regard to letter case. The table uses the vendor string when that vendor publishes one.
| Crawler | User-agent string | robots.txt token |
|---|---|---|
| AI2Bot | Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler) | AI2Bot |
| MistralAI-User | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots) | MistralAI-User |
| DeepSeekBot | DeepSeekBot in the Aiola record only | DeepSeekBot |
| Qwantbot | Mozilla/5.0 (compatible; Qwantbot/1.0_12345; +https://help.qwant.com/bot/) | Qwantbot |
Qwant's header can include -news and a worker identifier. Its documented examples retain the keyword Qwantbot.
How do you check the source address
A matching header does not prove who sent the request. Ai2's notice lists no addresses, and Aiola's directory records no IP ranges for AI2Bot. Mistral publishes four IPv4 prefixes for this fetcher. Those prefixes are 20.240.160.1/32, 20.240.160.161/32, 20.240.194.83/32 and 51.12.243.114/32.
Qwant recommends reverse DNS: check that the request address resolves to a hostname ending in qwant.com. Then confirm that hostname resolves back to the original address.
The help page also points to a JSON list of prefixes. Qwant's published file lists one IPv4 prefix, 194.187.171.0/24. DNS stays the recommended check, and the file is the alternative. DeepSeekBot has no ranges stored in the Aiola record.
Which robots.txt lines name them
Qwant documents full-site blocking with User-agent: Qwantbot and Disallow: /. Put robots.txt at each relevant subdomain's root.
Qwant's Crawl-delay: 8 example requests at least eight seconds between visits.
Ai2 says you can filter or reject traffic with the published user-agent string. The notice does not print a robots.txt group. Mistral uses named robots.txt tags, and MistralAI-User governs which sites a user request may visit. The same page says you can disallow MistralAI-Training, which is a separate token.
User-agent: AI2Bot
Disallow: /
User-agent: MistralAI-User
Disallow: /
User-agent: DeepSeekBot
Disallow: /
User-agent: Qwantbot
Disallow: /These groups use the directory's tokens to request full-site blocking. Ai2 documents header-based filtering, rather than promising compliance with this robots.txt example. Aiola's DeepSeekBot record provides no statement about robots.txt compliance.
Where the visits show in Analytics
The AI view shows crawl totals, bots, pages, and activity over time. It includes Answers, Indexing, Training and Other buckets, AI-answer referrals, crawler 404s, and discovery-file health.
MistralAI-User appears in Answers, and Qwantbot appears in Indexing. AI2Bot and DeepSeekBot appear in Training. These are Aiola's classifications, based on each directory record's kind. Kind fetch maps to Answers, kind index maps to Indexing, and kind crawl maps to Training. The DeepSeekBot Training label comes from Aiola's directory, which supplies no vendor documentation link.
Crawlers do not run JavaScript, so the browser tag cannot see them. One server or edge snippet from the Set Up Analytics posts each matching hit. Only known crawler user agents are stored.
A recorded crawl is not proof that the vendor cited the page. An empty period is not proof that the vendor cannot reach the site.
Try it in Aiola
Download Aiola free for macOS 12 or later and Windows 10 or later. Hosted Analytics needs Pro or the 7-day trial, and the pricing page lists both. Open the AI view to compare the four crawler names.
