What Is OAI-SearchBot, and Why Does ChatGPT Search Need It?

OAI-SearchBot: how it works (user agent, robots.txt marker and IP ranges)

OAI-SearchBot identifies itself in every request, checks robots.txt before fetching pages, and crawls from a published list of IP addresses. Those three facts, all from OpenAI’s crawler documentation, are what you need to recognise it, allow it and tell it apart from impostors that copy its name.

OpenAI’s Overview of OpenAI Crawlers gives this example string. OpenAI notes that the version number may change:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Notice the browser-like prefix. A filter that looks for “Chrome” and assumes a human visitor will misread this crawler.

When the bot fetches robots.txt itself, OpenAI says it may add a “robots.txt” marker before the link. That helps when your access logs store user agents but not request paths.

The IP ranges live at openai.com/searchbot.json. The copy we fetched on 26 October 2026 held 39 IPv4 prefixes and a creation time of 2 January 2026.

OAI-SearchBot vs GPTBot vs ChatGPT-User

OAI-SearchBot, GPTBot and ChatGPT-User share a company but do different jobs. OpenAI states that each setting is independent of the others. You can allow search, refuse training, and still let ChatGPT open a page when a user asks it to.

OpenAI adds that ChatGPT-User “is not used to determine whether content may appear in Search”. Search opt-outs belong to OAI-SearchBot alone.

There is also an efficiency rule. If a site allows both OAI-SearchBot and GPTBot, OpenAI may use one crawl for both purposes.

A fourth agent, OAI-AdsBot, only checks landing pages submitted as ChatGPT ads. Our GPTBot entry covers it alongside the training crawler.

OAI-SearchBot and ChatGPT search visibility

OAI-SearchBot access is the entry ticket to ChatGPT search, but it is only a ticket. Allowing the crawler makes your pages eligible. Whether ChatGPT then picks and cites them depends on how well the page answers the rewritten query and how trustworthy it looks.

A blocked site drops out of search answers, yet OpenAI says it “can still appear as navigational links”. Someone asking for your brand by name may still get a bare link.

What a blocked site loses is the richer placement: being quoted inside an answer, with a citation, when a buyer asks a category question.

Digital Magma treats this crawler as a commercial setting. A service firm, clinic or manufacturer gains almost nothing by hiding from ChatGPT search.

Our own robots.txt gives OAI-SearchBot a named group with the same exclusions as every other crawler. Only admin and login paths are closed.

For how ChatGPT turns a prompt into searches and citations, see how ChatGPT search works.

OAI-SearchBot robots.txt rules: allow search, block training

OAI-SearchBot rules take four lines when the goal is the common one: stay in ChatGPT search but keep pages out of model training. Each token below is spelled as OpenAI documents it. Choose one block, then copy your shared exclusions into it.

# A. Search yes, training no
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

# B. Search yes, one private folder closed
User-agent: OAI-SearchBot
Allow: /
Disallow: /client-portal/

# C. Out of ChatGPT search entirely
User-agent: OAI-SearchBot
Disallow: /

One rule from RFC 9309 catches people out. A crawler that finds its own group stops reading the * group.

So if your * group closes /wp-admin/ or /checkout/, repeat those lines inside the OAI-SearchBot group. Otherwise the named group quietly opens them.

Matching of the group name is case-insensitive under the RFC, so “oai-searchbot” works too. Digital Magma still copies OpenAI’s capitalisation so humans reading the file are not confused.

After saving, allow about 24 hours. OpenAI gives that figure for search systems to adjust to a robots.txt update.

How to verify OAI-SearchBot in logs and firewalls

Verifying OAI-SearchBot takes two pieces of evidence: requests that carry its token, and source addresses that sit inside searchbot.json. The name alone proves nothing, because any scraper can borrow it. The firewall check matters just as much, since a block there never shows up in robots.txt.

Start in the raw access log. On a typical Linux server:

grep "OAI-SearchBot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn

Each address in that list should fall inside a prefix from searchbot.json. Addresses outside the list are not OpenAI and can be blocked freely.

No hits at all over a few weeks is a warning sign. It often means something upstream is refusing the bot before it reaches your server.

So Digital Magma always checks the edge next. On Cloudflare, AWS WAF or a hosting firewall, filter security events by the user agent and read the action taken.

If you see blocks or challenges, add an allow rule that matches both the OAI-SearchBot token and the published IP prefixes. Matching both keeps impostors out.

Refresh those prefixes on a schedule. Digital Magma pulls the file fresh at every audit, because OpenAI’s list changes, and a hard-coded copy goes stale.

OAI-SearchBot in our client work

OAI-SearchBot access is the first item Digital Magma checks when a client asks why ChatGPT does or does not mention them. For a Canadian welding equipment manufacturer we have worked with for three years, ChatGPT visibility sits on top of 40+ keywords in Google’s top 10.

That pairing is the useful lesson. Strong pages that rank well on Google give ChatGPT search something worth citing once the crawler can reach them.

For a Toronto ENT clinic, the stakes are local. Patients ask assistants for a specialist near them, and a blocked crawler would remove the clinic from those answers.

For an app development company selling in six countries, the check runs per market. Each country section must load for the crawler, from the home page down to the deepest service page.

OAI-SearchBot mistakes that hide sites from ChatGPT

OAI-SearchBot mistakes usually come from tools acting on the site owner’s behalf. A plugin, a CDN preset or a copied robots.txt template blocks more than intended, and nobody notices because Google traffic stays normal while ChatGPT quietly stops citing the site.

• The “block AI bots” switch. Some CDN and plugin presets list OAI-SearchBot with the training bots. Read the generated rules, token by token.
• Treating OpenAI as one bot. A single “Disallow” for every OpenAI name blocks training and search together.
• Dropping shared exclusions. A named group with only “Allow: /” ignores the * group, so private paths open up.
• Hard-coding the version. Match “OAI-SearchBot”, never “OAI-SearchBot/1.4”. OpenAI says the number may change.
• Expecting overnight change. OpenAI gives about 24 hours. Judging the fix after one hour leads to wrong conclusions.
• Blocking by user agent alone. Real visits get stopped while spoofed ones slip through. Pair the token with the IP list.

One more limit is worth stating. RFC 9309 says robots.txt rules “are not a form of access authorization”. Pages that must stay private need a login.

OAI-SearchBot: frequently asked questions

OAI-SearchBot questions we hear from developers and site owners.

OAI-SearchBot: related terms, sources and author

OAI-SearchBot connects to these Digital Magma entries:

• GPTBot: OpenAI’s training crawler
• ChatGPT search: from query rewrite to citation
• Robots.txt for AI bots: rules for every major AI crawler
• AI crawler: the wider family of AI bots
• AI citation: what being cited is worth
• Why a website is not in ChatGPT

Sources: OpenAI’s Overview of OpenAI Crawlers and its searchbot.json list, and IETF RFC 9309. Both were checked on 26 October 2026.