Robots.txt rules for AI search, fetchers, and training bots

Choose which bot families may fetch public content and how that content may be used. Copy the relevant rules, preserve private-path exclusions, and verify requests after publishing.

Download AiolaSee pricing

Key facts

Choose search, answer, and training policies separately.
Robots.txt and Content-Signal express preferences; they are not authentication.
Specific user-agent groups need their own private-path exclusions.

Separate access from content use

Robots.txt asks cooperative crawlers which paths they may fetch. Content-Signal expresses permitted uses: search for indexing, ai-input for answering or retrieval, and ai-train for training. A permission signal does not override a blocked path, and a no signal is not a network access control. See the Content Signals specification for the vocabulary.

Aiola’s published robots.txt allows public crawling and all three uses, while excluding private routes. The examples below retain those exclusions when allowing a group. Choose your own policy; allowing discovery while declining training is a different decision from blocking every AI request.

Choose families and purposes

Find a family below, then retain its User-agent lines in the relevant blocks. Remove unneeded tokens. Selecting one identity does not select the entire vendor family.

Copy an allow or block policy

Copy one version per purpose. To make different family choices, split tokens into separate groups with their own directives. Leave a blank line between groups and customize private paths.

Search and indexing: allow or block

Allow public paths

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: MistralAI-Index
User-agent: xAI-SearchBot
User-agent: Google-CloudVertexBot
User-agent: Googlebot
User-agent: Applebot
User-agent: bingbot
User-agent: Amzn-SearchBot
User-agent: TikTokSpider
User-agent: Bytespider
User-agent: meta-webindexer
User-agent: YouBot
User-agent: Kimi-SearchBot
User-agent: Baiduspider
User-agent: Diffbot
User-agent: ImagesiftBot
User-agent: msnbot
User-agent: LinerBot
User-agent: QualifiedBot
User-agent: YandexBot
User-agent: DuckDuckBot
User-agent: SeznamBot
User-agent: PetalBot
User-agent: Qwantbot
Content-Signal: search=yes
Allow: /
Disallow: /dashboard
Disallow: /login
Disallow: /onboarding
Disallow: /auth/callback
Disallow: /api/
Disallow: /affiliate
Disallow: /demo

Block this group

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: MistralAI-Index
User-agent: xAI-SearchBot
User-agent: Google-CloudVertexBot
User-agent: Googlebot
User-agent: Applebot
User-agent: bingbot
User-agent: Amzn-SearchBot
User-agent: TikTokSpider
User-agent: Bytespider
User-agent: meta-webindexer
User-agent: YouBot
User-agent: Kimi-SearchBot
User-agent: Baiduspider
User-agent: Diffbot
User-agent: ImagesiftBot
User-agent: msnbot
User-agent: LinerBot
User-agent: QualifiedBot
User-agent: YandexBot
User-agent: DuckDuckBot
User-agent: SeznamBot
User-agent: PetalBot
User-agent: Qwantbot
Content-Signal: search=no
Disallow: /
User-triggered answers: allow or block

Allow public paths

User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
User-agent: Grok-DeepSearch
User-agent: Google-GeminiNotebook
User-agent: Google-Read-Aloud
User-agent: Google-Agent
User-agent: Copilot
User-agent: Amzn-User
User-agent: meta-externalfetcher
User-agent: DuckAssistBot
User-agent: Kimi-User
User-agent: Qwen-User
Content-Signal: ai-input=yes
Allow: /
Disallow: /dashboard
Disallow: /login
Disallow: /onboarding
Disallow: /auth/callback
Disallow: /api/
Disallow: /affiliate
Disallow: /demo

Block this group

User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
User-agent: Grok-DeepSearch
User-agent: Google-GeminiNotebook
User-agent: Google-Read-Aloud
User-agent: Google-Agent
User-agent: Copilot
User-agent: Amzn-User
User-agent: meta-externalfetcher
User-agent: DuckAssistBot
User-agent: Kimi-User
User-agent: Qwen-User
Content-Signal: ai-input=no
Disallow: /
Training and datasets: allow or block

Allow public paths

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: GrokBot
User-agent: xAI-Web-Crawler
User-agent: xAI-Grok
User-agent: xAI-Bot
User-agent: Grok
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: cohere-ai
User-agent: KimiBot
User-agent: QwenBot
User-agent: TongyiBot
User-agent: AliyunBot
User-agent: ERNIEBot
User-agent: YiyanBot
User-agent: ChatGLM-Spider
User-agent: DeepSeekBot
User-agent: AI2Bot
User-agent: Timpibot
User-agent: Doubaobot
Content-Signal: ai-train=yes
Allow: /
Disallow: /dashboard
Disallow: /login
Disallow: /onboarding
Disallow: /auth/callback
Disallow: /api/
Disallow: /affiliate
Disallow: /demo

Block this group

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: GrokBot
User-agent: xAI-Web-Crawler
User-agent: xAI-Grok
User-agent: xAI-Bot
User-agent: Grok
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: cohere-ai
User-agent: KimiBot
User-agent: QwenBot
User-agent: TongyiBot
User-agent: AliyunBot
User-agent: ERNIEBot
User-agent: YiyanBot
User-agent: ChatGLM-Spider
User-agent: DeepSeekBot
User-agent: AI2Bot
User-agent: Timpibot
User-agent: Doubaobot
Content-Signal: ai-train=no
Disallow: /
Advertising and previews: allow or block

Allow public paths

User-agent: Google-InspectionTool
User-agent: GoogleOther
User-agent: OAI-AdsBot
User-agent: meta-externalads
User-agent: facebookexternalhit
Allow: /
Disallow: /dashboard
Disallow: /login
Disallow: /onboarding
Disallow: /auth/callback
Disallow: /api/
Disallow: /affiliate
Disallow: /demo

Block this group

User-agent: Google-InspectionTool
User-agent: GoogleOther
User-agent: OAI-AdsBot
User-agent: meta-externalads
User-agent: facebookexternalhit
Disallow: /

The advertising and preview group has no AI Content-Signal, following Aiola’s current file. Purpose labels are Aiola’s classification, not proof of a vendor’s current behavior. Some entries are product-control tokens rather than a distinct crawler that will appear in request logs.

Allow discovery while declining training

Combine search allow, answer allow if desired, and training block. Keep the matching Content-Signal lines with each group, choosing yes or no to reflect your content-use permission.

For OpenAI’s documented separation, compare OAI-SearchBot, ChatGPT-User, and GPTBot with the official crawler guidance. Anthropic’s guidance likewise distinguishes its search, user, and training robots. User-triggered fetchers can have different rules; do not assume identical robots handling across vendors.

Publish without losing existing restrictions

Serve the file at /robots.txt on each relevant hostname. Merge these examples with your existing sitemap declaration and path policy. Specific user-agent groups need their own private-path exclusions; do not assume they inherit the wildcard group. Remove contradictory duplicate groups and verify the actual served file after deployment.

Tokens come from the directory; policy syntax follows Aiola’s file as of 21 October 2026. Detail pages retain per-record verification dates and sources. Tokens without official documentation have unconfirmed compliance.

Verify requests and observe the result

User-agent strings can be spoofed. Where available, follow the detail page’s published IP-range or reverse-DNS verification guidance. Robots exclusions do not authenticate requests or protect confidential content. Keep private resources behind application access controls and evaluate CDN or firewall behavior separately.

After publishing, compare server or edge logs with crawler analytics: requested paths, response statuses, and changes over time. Browser JavaScript alone misses many crawlers. Hosted Analytics requires Pro or trial. Observed crawls and assistant referrals do not prove a citation, recommendation, or compliance with every content-use preference.

Common questions

Build, ship and fix in one app

Errors, user feedback, GitHub and analytics live next to your coding agents in one desktop app, so every issue becomes a task an agent can take.

Download Aiola