OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for content that may train its models. ChatGPT-User may visit a page when a person asks ChatGPT or Custom GPTs. Search and training robots.txt settings are independent, per OpenAI's crawler documentation.
What each bot is for
OAI-SearchBot surfaces sites in ChatGPT search results. Sites that opt out are not shown in search answers. They can still appear as navigational links. OpenAI recommends allowing this bot in robots.txt. It also recommends allowing the published IP ranges.
GPTBot crawls content that may train OpenAI's generative models. Disallowing GPTBot says that content should not be used for that training. The version in its example user agent can change.
ChatGPT-User is not an automatic web crawl. It handles some user actions in ChatGPT and Custom GPTs. ChatGPT users may also interact with external applications through GPT Actions. Robots.txt rules may not apply, because a person started the request. This bot does not decide whether a page may appear in Search. Use OAI-SearchBot for search opt-out and for automatic crawl.
OAI-AdsBot only visits pages submitted as ads. OpenAI does not use that data to train generative models.
If you allow both OAI-SearchBot and GPTBot, OpenAI may reuse one crawl for search and training. ChatGPT search can take about 24 hours to follow a robots.txt update.
Match the full user agent
Start with the full User-Agent, the path, and the time. These strings come from the same OpenAI page.
Search example: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
OpenAI calls that an example, and the version may change. A robots.txt fetch may add a robots.txt marker before the URL. That marker separates a robots.txt fetch from other requests, even when a log has no path. GPTBot may add the same marker.
Training example: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
ChatGPT-User uses this full string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Keep each header with its path. Do not fold the three names into one OpenAI label. The crawler directory lists each identity with its sample and IP feed.
Allow search while blocking training
Copy this when public search crawling is allowed and training collection is not. The excluded paths are examples. Replace them with routes that exist on your site.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /api/Merge the example into your current file. Keep sitemap lines and unrelated groups. The robots.txt guide says a specific group does not inherit wildcard blocks. Give OAI-SearchBot its own private-path lines.
This example leaves out ChatGPT-User, because robots.txt may not apply to it. OpenAI's documented controls for this split are the two tokens above. The same guide keeps Content-Signal use lines separate from path rules. A use line does not override a blocked path. It is not a network control.
Serve the file at /robots.txt on each hostname you mean to cover. Open the public URL after deployment and compare it with the text you saved.
A robots.txt rule asks a cooperative crawler which paths it may fetch. It does not protect private pages. Keep those pages behind application access controls. The guide also says a user-agent string can be spoofed. Check CDN or firewall rules as their own control.
Compare addresses with the right feed
OpenAI publishes feeds for OAI-SearchBot, GPTBot, and ChatGPT-User. Each feed includes creationTime and a prefixes list. Each entry is an ipv4Prefix range. Take the source address from your server or edge log. Compare it with the feed for the bot named in the header.
The lists are not the same. In the feeds checked for this update, six prefixes were in both the GPTBot and OAI-SearchBot lists. One shared prefix was 4.227.36.0/25. ChatGPT-User shared no prefixes with those two lists in this check. The prefix 132.196.86.0/24 was only in the GPTBot feed in this check.
A shared prefix does not show which bot sent the request. A feed entry is a range, so the request address can differ from the range's first address. Keep the user agent beside the matching range. Use the GPTBot verification guide for the step-by-step range check.
Read the three bots in Analytics
Bot crawlers puts OAI-SearchBot in Indexing, GPTBot in Training, and ChatGPT-User in Answers. OAI-AdsBot appears in Other.
The AI view shows crawl totals, bots, pages, and activity over time. It groups hits into Answers, Indexing, Training, and Other. AI-answer referrals, crawler 404s, and discovery-file health also appear there.
Crawlers do not run JavaScript, so the browser tag cannot see them. One server or edge snippet from the setup guide posts each matching hit. Only known crawler user agents are stored.
A recorded crawl is not proof that the vendor cited the page. An empty period is not proof that the vendor cannot reach the site.
Try it in Aiola
Download Aiola free for macOS 12 or later and Windows 10 or later. Hosted Analytics needs Pro or the 7-day trial, and the pricing page lists both. Open the AI view and compare the three OpenAI crawler rows.
