Meta-ExternalAgent, Meta-ExternalFetcher, and Meta-WebIndexer do different jobs. A meta-externalagent request names that one crawler. It does not stand for every Meta hit on your site.
What each crawler is for
Meta's web-crawlers page gives each crawler a purpose and sample headers. The last column is Aiola's bucket for that identity.
| Crawler | Meta's documented purpose | Aiola Analytics bucket |
|---|---|---|
| Meta-ExternalAgent | Trains foundation AI models, or improves products by indexing content directly | Training |
| Meta-ExternalFetcher | Fetches individual links a user requested, and evaluates agentic AI that navigates sites to complete tasks | Answers |
| Meta-WebIndexer | Improves Meta AI search quality. Allowing it in robots.txt helps Meta cite and link to your content | Indexing |
Meta documents two more crawlers on that page. Meta-ExternalAds crawls for advertising and other business products. FacebookExternalHit collects a shared page's title, description, and thumbnail. It serves link previews on Facebook, Instagram, and Messenger.
Aiola places meta-externalads and facebookexternalhit under Other. It places FacebookBot with Meta-ExternalAgent under Training. A hit on one of those names is not a WebIndexer or ExternalFetcher visit.
ExternalFetcher may bypass robots.txt when a user requested the fetch. FacebookExternalHit might bypass robots.txt for security or integrity checks. Meta states those exceptions separately. A disallow for one token does not cover the other tokens.
Choose the rule for each purpose separately. A training preference for ExternalAgent does not state a search preference for WebIndexer. Keep the ExternalFetcher group separate as well.
The ExternalAgent token covers training and direct indexing together. Meta does not publish a second token that splits those two uses. Crawler preferences go in robots.txt, not NoAI tags, on Meta's page.
Match the sample user agents
Meta says a log header will be similar to one of two samples. ExternalAgent's samples are meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers) and meta-externalagent/1.1.
ExternalFetcher's samples are meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers) and meta-externalfetcher/1.1. WebIndexer's samples are meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers) and meta-webindexer/1.1.
The robots.txt token drops the version and the parenthetical path. Meta's example uses User-agent: meta-externalagent, not the longer sample.
Aiola matches Meta-ExternalAgent, meta-externalagent, and FacebookBot to Training. It matches Meta-ExternalFetcher or meta-externalfetcher to Answers. It matches meta-webindexer to Indexing. The Set Up Analytics guide matches these tokens with a case-insensitive pattern.
Aiola's crawler directory records AS32934 for each of these three.
Give each token its own robots.txt group
Meta says to add a disallow for the crawler you want to block. Its sample puts Allow: / and Disallow: /private/ in the ExternalAgent group. It shows the same shape for meta-externalads.
This example allows WebIndexer outside /private/, blocks ExternalAgent, and disallows ExternalFetcher.
User-agent: meta-webindexer
Allow: /
Disallow: /private/
User-agent: meta-externalagent
Disallow: /
User-agent: meta-externalfetcher
Disallow: /Replace /private/ with a directory you want to exclude. To block the search crawl, put Disallow: / in the WebIndexer group. To permit ExternalAgent, use Allow: / and add path exclusions in that group.
Meta says ExternalFetcher may still bypass robots.txt for user-requested fetches. A disallow does not promise those fetches will stop.
The robots.txt guide says to merge groups without assuming wildcard inheritance. That page also says to keep your sitemap line and path rules. It says to remove contradictory duplicate groups.
Plan for robots.txt caching
Meta asks you to allow up to 24 hours for a robots.txt change to take effect. Crawlers may cache the file for up to 24 hours. A request during that window can still follow the older copy.
After deployment, open /robots.txt on the hostname you changed. Meta lists [email protected] for questions about these crawlers.
Send hits to the crawler endpoint
Crawlers do not run JavaScript, so the browser tag cannot see them. One server or edge snippet from the setup guide posts each matching hit. Only known crawler user agents are stored.
The Analytics documentation describes the AI view.
Read the four buckets in the AI view
The AI view shows crawl totals, bots, pages, and activity over time. It groups hits into Answers, Indexing, Training, and Other. AI-answer referrals, crawler 404s, and discovery-file health also appear there.
Meta-ExternalAgent appears in Training, Meta-ExternalFetcher in Answers, and Meta-WebIndexer in Indexing.
A recorded crawl is not proof that Meta cited the page. An empty period is not proof that Meta cannot reach the site.
Try it in Aiola
Download Aiola free for macOS 12 or later and Windows 10 or later. Hosted Analytics needs Pro or the 7-day trial, and the pricing page lists both. Open the AI view and compare the three Meta crawler rows.
