AI search glossary
Robots.txt for AI Bots, Explained With 4 Rule Sets You Can Copy
A CDN toggle or a vendor’s advice to “block all AI” can remove you from ChatGPT, Claude and Perplexity search. This entry lists every major AI token from its owner’s documentation and gives four tested rule sets.
Robots.txt for AI bots: the short answer
Robots.txt for AI bots works one user-agent token at a time. Each AI company publishes separate tokens for model training, search indexing and user-requested fetches, so you can block training bots, allow search bots, or set both. Rules are requests, not locks, and some user-triggered fetchers may not apply them at all.
Key takeaways
- Decide training and search separately. OpenAI, Anthropic and Perplexity each use different tokens for them.
- A named group replaces the * group, so repeat your private paths inside it.
- Google-Extended controls Gemini use only. AI Overviews follow Googlebot rules.
- ChatGPT-User and Perplexity-User may ignore robots.txt because a person started the fetch.
- CDN and firewall rules can block what robots.txt allows.
Robots.txt rules for AI crawlers: how they work
Robots.txt rules for AI crawlers follow the same standard as every other bot: IETF RFC 9309, the Robots Exclusion Protocol. A crawler looks for a group naming its token, obeys that group’s allow and disallow lines, and falls back to the * group only when no named group exists.
RFC 9309 sets the details that matter most:
- Case. Crawlers must match the group name case-insensitively.
- Fallback. With no matching group, the crawler obeys the * group.
- Precedence. The most specific path match wins, meaning the longest one.
- Errors. If robots.txt is unreachable through server errors, crawlers must assume complete disallow.
- Caching. Crawlers should not use a cached copy for more than 24 hours.
The fallback rule causes the classic AI mistake. Add “User-agent: GPTBot” with “Allow: /”, and GPTBot stops reading your * group entirely.
Your admin, cart and login exclusions vanish for that bot. Every named group needs its own copy of them, which is why Digital Magma writes the shared block once and pastes it into each group.
Google’s introduction to robots.txt covers basic syntax and placement at the site root.
Robots.txt tokens for AI: training, search and user fetchers in one table
Robots.txt tokens for AI fall into three jobs, and grouping them that way makes every decision easier. Training bots gather content for future models. Search bots index pages so AI answers can cite them. User fetchers open a page live when someone asks. Each row below comes from the owner’s own documentation.
| Company | Training | Search | User fetcher |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User (robots.txt “may not apply”) |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User (site owners can disable it) |
| Perplexity | None named. PerplexityBot is “not used” for foundation models. | PerplexityBot | Perplexity-User (“generally ignores robots.txt”) |
| Google-Extended (also grounding) | Googlebot, which covers AI Overviews and AI Mode | Not applicable |
Sources: OpenAI’s crawler overview, Anthropic’s crawling help article, Perplexity Crawlers and Google’s common crawlers page.
OpenAI also runs OAI-AdsBot, which only visits landing pages submitted as ChatGPT ads. Most sites can leave it alone.
Anthropic is the outlier on user fetches. Its article says disabling Claude-User stops retrieval for user queries, so that group does take effect.
Robots.txt for AI bots: four rule sets you can copy
Robots.txt for AI bots comes down to four common positions: allow everything, block training but keep search, block all AI, or close one section. Each set below uses only tokens the owners document. Copy one set, add your own shared exclusions to every group, and save it at the root.
# SET 1: allow all AI bots (with shared exclusions)
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /
Disallow: /wp-admin/
# SET 2: block training, keep AI search
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /wp-admin/
# SET 3: block all documented AI crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
Disallow: /
# SET 4: close one section to every AI token
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
Disallow: /research/Set 2 suits businesses that worry about training. It keeps ChatGPT, Claude and Perplexity search open.
Set 3 still cannot stop ChatGPT-User or Perplexity-User. Their owners say robots.txt may not apply to user-started visits.
No set touches Googlebot, and Digital Magma never adds it to an AI block. Blocking it removes you from Google Search, AI Overviews included, which is a separate and drastic decision.
Digital Magma uses a fuller version of Set 1 on its own robots.txt, with identical exclusions in every named group.
Robots.txt for AI bots: how to test
Robots.txt for AI bots needs testing in two places: the file itself and the traffic that follows. A valid file proves the syntax is right. Only server logs prove that each crawler obeys it, and that nothing upstream blocks the bots you meant to allow.
- Fetch the live file. Open yoursite.com/robots.txt on every host, including subdomains.
- Check syntax. Search Console’s robots.txt report shows how Google read the file and flags errors.
- Check size. Google enforces a 500 KiB limit and ignores anything after it.
- Read the logs. Filter by each token and confirm the source IPs match the owner’s published lists.
- Wait the documented time. OpenAI gives about 24 hours for search. Google caches for up to 24 hours.
Published IP lists: openai.com/gptbot.json and searchbot.json, claude.com/crawling/bots.json, and perplexity.com/perplexitybot.json.
Digital Magma recommends saving a dated copy of the file before every edit. If traffic drops after a change, the old version can go back in minutes.
Robots.txt limits: what it cannot do for AI
Robots.txt cannot secure anything, cannot remove content already collected, and cannot bind fetchers that choose not to follow it. RFC 9309 says the rules “are not a form of access authorization”. For AI, those limits are sharper, because some user-triggered fetchers say openly that they may skip the file.
- It is not a lock. Private pages need a login. Anthropic notes its bots will not try to bypass CAPTCHAs.
- It looks forward. Anthropic describes a ClaudeBot block as covering “future materials”.
- It does not deindex. Google says robots.txt is not a way to keep a page out of Google. Use noindex or a password.
- It may not reach user fetchers. ChatGPT-User and Perplexity-User both carry that caveat.
- Crawl-delay is patchy. Anthropic supports it. Google’s specification says Google does not.
A noindex rule needs the page to be crawlable. Block a page in robots.txt and Google never sees its noindex tag.
Robots.txt, CDN bot settings and firewalls
Robots.txt only speaks to bots that reach your server, and many requests never get that far. CDN bot presets, managed AI-blocking rules and web application firewalls can refuse an AI crawler before robots.txt matters. When AI visibility drops while the file looks correct, the edge is the first place to check.
Perplexity’s documentation even includes allow-rule recipes for Cloudflare and AWS WAF. It advises matching the user agent and the published IP range together.
OpenAI asks sites to allow requests from its published IP ranges for OAI-SearchBot. A strict bot setting can quietly break that.
Anthropic warns the other way too. Blocking its IPs instead of using robots.txt “may not work correctly or persistently guarantee an opt-out”.
So keep each layer in its lane. Robots.txt states your policy, the firewall stops impostors, and logs prove both.
Digital Magma reads the CDN bot settings in the same session as the robots.txt file. Reviewing one without the other is how sites end up blocking the crawlers they meant to welcome.
Robots.txt for AI in our client work
Robots.txt for AI is now a standing item in every Digital Magma technical audit. For a Canadian welding equipment manufacturer visible in ChatGPT and Google’s AI Overviews over a three-year engagement, the point of the file is simple: keep every search bot open and the admin paths closed.
That manufacturer’s 40+ top-10 keywords are the base AI answers draw on. A careless AI block would put that visibility at risk.
For a Toronto ENT clinic, robots.txt sits next to stricter security. Patient-facing pages stay open while booking and portal paths stay closed.
For an app development company in six countries, each country host and subdomain carries its own file, kept identical in policy.
Robots.txt for AI: frequently asked questions
Robots.txt for AI questions we answer most often.
How do I block AI bots with robots.txt?
Add a group per token, such as GPTBot, ClaudeBot and Google-Extended, with “Disallow: /”. Decide separately whether to block search tokens like OAI-SearchBot, Claude-SearchBot and PerplexityBot.
How do I disallow GPTBot but allow OAI-SearchBot?
Write two groups: “User-agent: GPTBot” with “Disallow: /”, then “User-agent: OAI-SearchBot” with “Allow: /”. OpenAI says the two settings are independent.
Should I block AI training bots?
It depends on whether your content is the product. Publishers that license archives often block training. Service firms that want accurate AI descriptions usually allow it.
Can robots.txt block AI agents and scrapers?
Only those that choose to obey it. RFC 9309 says the rules are not access authorization. Use a login for private content and a firewall for abusive traffic.
Does blocking AI bots hurt SEO?
Blocking training tokens does not affect Google Search, and Google says Google-Extended is not a ranking signal. Blocking Googlebot does, because it removes you from Search.
Do AI crawlers support Crawl-delay?
Anthropic says ClaudeBot supports the non-standard Crawl-delay line. Google does not support it. OpenAI’s and Perplexity’s crawler pages do not mention it.
How long do AI crawlers take to read robots.txt changes?
OpenAI gives about 24 hours for search. Google caches the file for up to 24 hours, and RFC 9309 says crawlers should not cache it longer.
Robots.txt for AI that keeps search open
Digital Magma writes and tests robots.txt for AI crawlers, then checks CDN rules and logs so the policy holds. See our AI SEO service or book a free SEO audit.
No AI citation is guaranteed. Call +91 79823 19697, email contact@digitalmagma.com or message on WhatsApp (Mon–Sat, 9 am–9 pm IST). Read our Google reviews any time.