Check a GPTBot request against OpenAI's published IP ranges before you allow it. You can block GPTBot for training and choose search access separately. Treat the claimed user agent, your robots.txt policy, and network access as separate checks.
Match the name, then check the source
OpenAI publishes this exact example string in its crawler documentation:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
The version can change. Robots.txt fetches may add a robots.txt marker. Use GPTBot as the robots.txt token. Those details come from the same OpenAI page.
Start with one real request. Keep its timestamp, path, status, full user agent, and source IP together.
A user-agent string can be spoofed. A matching name starts the check and does not finish it. Do not allow a request only because the text matches.
Aiola's GPTBot reference puts its identity, Training classification, documentation, and IP feed in one place. Use the vendor page for current behavior.
Compare the address with published ranges
OpenAI publishes separate JSON feeds for GPTBot, OAI-SearchBot, and ChatGPT-User. Each feed has a prefixes array of ipv4Prefix values.
The GPTBot feed checked for this update includes 132.196.86.0/24. That prefix was not in the other two feeds. It is one entry, not the full list. Read the current GPTBot feed for the request you have.
Use this sequence:
- Take the source address from your server or edge log.
- Open the feed for the identity that request claims.
- Test the address against each CIDR, not as a text search.
- Record the matching range and the time you checked.
- Hold requests with no match for a separate review.
A published entry is a range. The request address can differ from that range's first address.
Some prefixes appear in both the GPTBot and OAI-SearchBot feeds. For example, 4.227.36.0/25 was in both feeds checked for this update. A shared prefix means OpenAI published that address for more than one bot. It does not show which bot sent the request. The ChatGPT-User feed shared no prefixes with those two lists in the same check.
Save the request and the feed entry before you change a firewall rule.
Do not treat an Analytics bot name as proof of identity. Aiola classifies the bot from the user agent. Use server or edge logs for the address check.
Choose training, search, and fetches separately
OpenAI documents three identities for this choice. GPTBot crawls content that may be used to train OpenAI models. Disallowing GPTBot indicates that content should not be used for that training. OAI-SearchBot is for ChatGPT search. Opting out removes the site from search answers, while navigational links can remain. ChatGPT-User fetches a page after a person asks. Robots.txt may not apply to those user-triggered fetches. These points come from OpenAI's crawler guidance.
OpenAI also documents OAI-AdsBot for pages submitted as ads. Aiola places that bot in Other, not in Training.
If you allow both GPTBot and OAI-SearchBot, OpenAI may reuse one crawl for training and search.
A network block stops user-triggered fetches from reaching the blocked pages. Robots.txt does not do that.
Keep private pages behind application access controls. A robots.txt preference does not replace authentication.
Block training while allowing search
This example blocks GPTBot and allows OAI-SearchBot on public paths:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Disallow: /dashboard
Disallow: /login
Disallow: /api/To allow GPTBot, replace its Disallow: / line with Allow: /. Then add your private-path lines to that group. To block search, replace the search group's lines with Disallow: /.
Use your real private paths. The three exclusions above are examples.
Do not assume that a wildcard group's rules carry into a specific group. Give each named group its own exclusions. Keep existing sitemap lines and unrelated rules when you merge this example.
Aiola's robots.txt guide separates purpose groups from content-use preferences. Keep path rules and content-use lines distinct.
Serve the file at /robots.txt on each hostname you mean to cover. Read the served file after you deploy it.
Check what changed after publishing
Pick a few public paths before you deploy. Include an article, a docs page, and one public path you mean to exclude.
Save the old policy and the deployment time. Then compare logs across similar periods. Check identities, paths, statuses, and whether they match the choice you made.
OpenAI says ChatGPT search can take about 24 hours to follow a robots.txt change.
A robots.txt edit states a crawl preference. A firewall rule decides whether the request can reach the content. Review those controls separately.
If the result looks wrong, read the served robots.txt first. Then check that user-agent group and your network rules on their own.
A Training row means a training crawler requested that URL. It does not prove a citation or a live answer. An assistant referral is observed assistant traffic. It does not prove that a vendor cited the page.
Review crawler activity in Aiola
Aiola is one desktop app for macOS 12+ and Windows 10+. Aiola Analytics names 70 crawlers, including GPTBot, OAI-SearchBot, and ChatGPT-User.
The AI view shows bots, pages, crawler 404s, and change over time. It places GPTBot in Training, OAI-SearchBot in Indexing, and ChatGPT-User in Answers.
Crawler hits reach Analytics from a server or edge call. AI crawlers do not run JavaScript, so the browser tag cannot see them. Only known crawler user agents are stored.
Analytics does not store the source address. It stores the path, user agent, optional status, and optional country. A stored 200 can mean the capture omitted status. Confirm the address in your server or edge logs.
No row means Aiola received no matching beacon in that period. It does not prove the vendor cannot access the site.
Open the project for the site you changed. Compare its paths with the list you saved. Use crawler analytics next to those logs.
Hosted Analytics needs Pro or the seven-day trial. The trial is for a first-time subscriber on Pro Monthly at 100,000 events. A card is collected at checkout.
Try it in Aiola: download the desktop app for free and review crawler activity next to your logs. Hosted Analytics needs Pro or that same trial.
