What Is GPTBot, and Should Your Robots.txt Allow It?

GPTBot: how it works (user agent, robots.txt marker and IP list)

OpenAI’s training crawler behaves like other well-mannered bots. It names itself in the user-agent string, reads robots.txt, and fetches only what the rules allow.

OpenAI’s Overview of OpenAI Crawlers gives this example user-agent string, and warns that the version number may change:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

When it fetches robots.txt, the crawler may add a “robots.txt” marker to that string. OpenAI says this helps owners spot robots.txt requests when their logs leave out the path.

The crawler’s IP ranges are published at openai.com/gptbot.json. The copy we fetched for this entry listed 18 IPv4 prefixes, with a creation time of 22 September 2026.

The list changes, so read it fresh each time rather than copying ranges into a firewall once.

GPTBot vs OAI-SearchBot vs ChatGPT-User vs OAI-AdsBot

GPTBot is one of four OpenAI agents. Each setting is independent of the others. The table follows OpenAI’s crawler page, checked on 21 October 2026.

OpenAI adds that if you allow both the training and search bots, it may use one crawl for both, to avoid crawling twice.

Two more details matter. OAI-SearchBot-blocked sites can still show as navigational links. OAI-AdsBot data, OpenAI says, is not used to train foundation models.

GPTBot for a business: why the training decision matters

GPTBot decides one thing only: whether your public pages may feed future OpenAI models. That is a business choice, and it deserves a deliberate answer.

A publisher that licenses its archive may block training. A service firm that wants its name, prices and expertise widely known may allow it, so later models describe it accurately.

Digital Magma takes the second view for its own site. Our robots.txt allows all four OpenAI agents, each in its own named group, with the same admin exclusions repeated in every group.

The costly mistake is mixing the decisions up. A site that blocks “all AI bots” in one sweep stays out of training, and out of ChatGPT search answers as well.

Other AI companies split training the same way. Anthropic’s help article says restricting ClaudeBot signals that future material should be left out of its training data.

Google handles it with a control token. Google-Extended has no user-agent string of its own. It exists only as a robots.txt token, so blocking it does not stop Googlebot.

So write the two answers down separately: training yes or no, and search yes or no. Then set the groups to match.

Digital Magma recommends starting every crawler audit with these two questions, because the answers settle every rule that follows.

GPTBot: what to do (robots.txt rules you can copy)

Each rule below uses the tokens OpenAI documents. Pick the block that matches your decision.

# 1. Block training, stay in ChatGPT search
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

# 2. Allow training, but keep one folder out
User-agent: GPTBot
Allow: /
Disallow: /members/

# 3. Allow everything for both
User-agent: GPTBot
User-agent: OAI-SearchBot
Allow: /

Then work through four checks, in this order:

1. Place the file at the root. Crawlers only read /robots.txt at the top of each host.
2. Copy your shared exclusions. Under RFC 9309, a crawler with its own group ignores the * group. Admin and login paths must be repeated.
3. Check the CDN and firewall. Allow OpenAI’s published IP ranges, or a bot rule may block what robots.txt allows.
4. Wait, then verify. OpenAI says search changes take about 24 hours. Then confirm in your logs.

Google’s robots.txt introduction covers the general syntax if you are new to the file.

A note on matching. RFC 9309 says crawlers must match the user-agent group without regard to case, so “gptbot” and “GPTBot” both work. Digital Magma still writes the token exactly as OpenAI does, for readability.

How to verify GPTBot in server logs

Verifying GPTBot in logs takes two matches: the user-agent string and the source IP. Anyone can fake the name, so the name alone proves nothing.

Start by pulling the matching lines from your access log. On most Linux hosts this works:

grep "GPTBot" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn

That lists each IP that used the name, with a count. Next, check each IP against the prefixes in gptbot.json.

An address inside a listed prefix is a real OpenAI crawl. Anything outside is an impostor, and you can block it without touching robots.txt.

On a CDN such as Cloudflare, use the bot or firewall event log instead. It shows the user agent, the IP and whether a rule blocked the request.

When Digital Magma audits a site, this log check comes before any robots.txt edit. It shows what is really happening, not what the file intends.

GPTBot mistakes: blocking search by accident, CDN rules and noindex confusion

Most GPTBot mistakes come from treating OpenAI as a single bot, or robots.txt as the only control.

• Blocking search by accident. A “block AI bots” toggle or plugin may include OAI-SearchBot. Read the generated file and check which tokens it lists.
• Forgetting the CDN. OpenAI’s help article on ChatGPT search asks owners to confirm the host or CDN allows its published IP addresses.
• Losing shared rules. A named group with only “Allow: /” drops every Disallow from the * group for that crawler.
• Expecting noindex to work. OpenAI’s crawler page names robots.txt tokens as the control. It does not mention noindex.
• Blocking the page you want removed. Google warns that a robots.txt block stops its crawler from ever seeing a noindex rule.
• Hard-coding the version. Match “GPTBot”, not “GPTBot/1.4”. OpenAI says the number may change.

Treat robots.txt as a request. RFC 9309 says its rules “are not a form of access authorization”. Private content needs a login.

GPTBot: frequently asked questions

Short answers to the GPTBot questions site owners ask Digital Magma most.

GPTBot: related terms, sources and author

GPTBot connects to these entries in the Digital Magma knowledge base:

• OAI-SearchBot: OpenAI’s crawler for ChatGPT search
• AI crawler: the bots that fetch pages for AI systems
• Robots.txt for AI: rules for every major AI user agent
• ChatGPT search: how ChatGPT finds and cites pages
• LLM optimization: shaping how language models describe you
• Should you block or allow AI crawlers? and why a website is not in ChatGPT

Sources: OpenAI’s Overview of OpenAI Crawlers and its IP lists, OpenAI’s ChatGPT search help article, IETF RFC 9309, and Google Search Central on robots.txt and noindex.

All were checked on 21 October 2026. Version numbers and IP ranges change, so recheck OpenAI’s page before you edit robots.txt.