Meta-WebIndexer crawls pages to improve Meta AI search results and support citations and links. Allow it when you want that use of your public content. Configure Meta's training crawler and user-requested fetcher separately.
What does Meta-WebIndexer fetch?
Meta's web crawlers page describes Meta-WebIndexer as a crawler for search quality. It analyzes online content to improve relevance and accuracy. Meta says allowing it helps Meta AI cite and link to your content.
That purpose differs from model training. Meta documents training under Meta-ExternalAgent, which also performs direct indexing. Choose which crawler to allow by its documented uses, rather than treating every Meta request alike.
Aiola's Meta-WebIndexer record lists its category as Indexing and its kind as search. Its notes describe search results, citations, and links. The record provides the sample user-agent and robots.txt token for comparison with incoming requests.
FacebookExternalHit has another purpose: gathering shared-page metadata for link previews. Its requests concern previews on Facebook, Instagram, and Messenger. A link-preview request therefore represents a different activity from a Meta AI search crawl.
What do the other two crawlers fetch?
Meta-ExternalAgent crawls content for foundation-model training and direct indexing to improve Meta products. Those purposes share the same documented crawler token. Allowing its token covers both uses; the documentation provides no separate token for each purpose.
Aiola's Meta-ExternalAgent record categorizes it as Training. That category summarizes the crawler's purpose without changing Meta's description of its additional indexing use.
Meta-ExternalFetcher retrieves individual links at a user's request. It also supports evaluating and improving agentic AI capabilities, including navigating websites to complete tasks. Meta says this crawler may bypass robots.txt rules.
Aiola's Meta-ExternalFetcher record categorizes it as AI answers. Its kind is fetch, which maps to the Analytics dashboard's Answers bucket.
Meta also documents Meta-ExternalAds for advertising and other business products. Its separate token is meta-externalads. FacebookExternalHit may bypass robots.txt during security or integrity checks. Meta states that exception separately from the user-requested fetcher's exception.
Which user-agent string should you match?
Meta publishes two sample forms for Meta-WebIndexer. The longer form is meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers). The shorter form is meta-webindexer/1.1.
Use meta-webindexer as the robots.txt token, without the version or parenthetical path. Aiola's crawler record stores that token separately from the longer sample user-agent.
Meta-ExternalAgent follows the same pattern. Its samples are meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers) and meta-externalagent/1.1. Its robots.txt token is meta-externalagent.
Meta-ExternalFetcher uses meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers) or meta-externalfetcher/1.1. Its robots.txt token is meta-externalfetcher.
Compare the complete incoming header with those samples before selecting a robots.txt group. Keep the sample string distinct from the shorter token used in the file. A versioned sample helps identify a log entry; it is not the token shown in Meta's robots.txt example.
Aiola's three records list AS32934 and a verification date of 2026-10-23. These are directory fields, separate from the request fields documented for Analytics ingestion.
How do you allow or block each one?
Meta instructs site owners to add a disallow for the relevant crawler. Its example separates crawler groups and combines a root allow with a directory disallow.
This example allows Meta-WebIndexer outside /private/, blocks Meta-ExternalAgent, and states a disallow for Meta-ExternalFetcher:
User-agent: meta-webindexer
Allow: /
Disallow: /private/
User-agent: meta-externalagent
Disallow: /
User-agent: meta-externalfetcher
Disallow: /Replace /private/ with a directory relevant to your site. Add further directory exclusions inside the allowed group as needed. To decline the search crawl entirely, replace its allow and directory exclusion with Disallow: /.
The Meta-ExternalAgent group covers training and direct indexing together. To permit both, replace its root disallow with Allow: / and add any directory exclusions.
The fetcher group expresses a preference with the bypass limitation Meta documents. It does not guarantee that user-requested fetches will stop.
Merge the chosen groups into your existing robots.txt file. Check the published file on the hostname receiving the requests. Meta says changes may take up to 24 hours because crawlers can cache the file.
How does Aiola group them?
Aiola Analytics names 70 crawlers. Its AI view shows crawl totals, bots, requested pages, timeseries, and four purpose buckets. Search and classic indexing map to Indexing; crawl maps to Training; fetch maps to Answers. Advertising and social kinds map to Other.
Crawler collection requires setup on your website. The browser tracking tag alone cannot collect these crawler visits. Follow Set Up Analytics and choose one server or edge implementation for page requests.
In the project, open Settings → Analytics, save the production hostname, and enable Analytics if needed. Enabling creates a public Tracking ID. The saved domain is required; Analytics does not use a secret key.
Crawler reports include the Tracking ID, requested path, hostname, and user-agent. Country and response status are optional fields. Only known crawler user-agents are stored, and the hostname must match the configured domain.
What does a robots.txt rule not do?
A robots.txt rule describes your crawler preference. Meta's documented bypass exceptions mean that a disallow cannot guarantee the absence of every Meta request.
After publishing, allow the documented cache window before comparing requests. Check the affected token and directory in your server logs. In Aiola's AI view, review the reported bots, pages, and crawl timeseries after installing crawler tracking.
Crawler activity and AI-answer referrals are separate items in that view. A recorded crawl shows a request; it does not establish that a particular response cited your page.
Try it in Aiola: download the desktop app for macOS 12+ or Windows 10+. Hosted Analytics requires Pro or the seven-day trial. The trial is for first-time Pro Monthly customers and requires a card.
