Should I Block AI Crawlers, or Allow Them? Training vs Search Bots

What are the three kinds of AI crawlers?

AI crawlers fall into three groups by job: training bots collect pages for future models, search bots index pages so an assistant can cite them, and user fetchers open a page because someone asked. Each AI company now publishes a separate name for each job.

That split is the most useful fact on this page. A single “AI bots” switch hides it.

Why does it matter so much? Because the three jobs carry three different costs and benefits. Training use is about the future. Search use is about this week’s buyers. User fetches are about one person, right now, who asked an assistant to read your page.

Digital Magma writes the three decisions in a short table before touching any file. Training: yes or no. Search: yes or no. User fetches: usually yes, since you rarely control them anyway.

The sources behind the table are the owners’ own pages:

• OpenAI. The Overview of OpenAI Crawlers says each setting is independent. You can allow OAI-SearchBot for search while disallowing GPTBot for training.
• Anthropic. Its help article on crawling says restricting ClaudeBot signals that future material should be left out of training. Disabling Claude-SearchBot may reduce visibility in search results.
• Perplexity. Its crawler documentation says PerplexityBot surfaces sites in search and is not used to crawl for AI foundation models.
• Google. The Google common crawlers page says Google-Extended has no user agent of its own. It is a robots.txt token used for control.

When should you allow AI crawlers?

Allow AI crawlers for search whenever you want buyers to find you through ChatGPT, Perplexity, Claude or Copilot. That covers most service firms, clinics, manufacturers, shops and agencies. For them, being described accurately by an assistant is worth more than keeping public pages away from it.

Ask a plain business question. Does a buyer gain anything when an AI assistant reads your pricing page, your service list or your FAQ? For most companies Digital Magma works with, the answer is yes.

Three situations make allowing AI crawlers the obvious choice:

• You sell a service or product. Your pages exist to be read and quoted. Hiding them from search bots hides them from buyers who now ask assistants first.
• You compete on expertise. Guides, case results and clear prices give an assistant something specific to cite.
• Your brand is often misdescribed. Letting search bots read the current facts gives answers a better source than old directory listings.

Allowing training bots is a softer call. Digital Magma allows GPTBot and ClaudeBot on its own site, so future models learn our name and services correctly. A firm with nothing to sell from its archive may choose the same.

When does blocking AI crawlers make sense?

Blocking AI crawlers makes sense when your content is the product: paid archives, licensed research, original data, course material or anything you sell access to. In those cases blocking training bots protects value. Blocking search bots too is a real trade-off, because it also removes your citations.

Publishers carry the strongest case. An archive of reporting or analysis has licensing value, and some publishers negotiate that value directly with AI companies.

A training block still leaves room for citations. A publisher can say no to model training and yes to being linked as a source in answers. That keeps referral traffic while protecting the archive.

Other sound reasons to block some AI crawlers:

• Members-only areas. Disallow paths such as /members/ or /portal/ for every bot, and protect them with a login as well.
• Server load. If a crawler hits a small server hard, Anthropic supports the non-standard Crawl-delay rule for its bots. That slows the crawl without a full block.
• Legal or client limits. Some contracts forbid machine reuse of certain documents.

Digital Magma also sees blocks set up in a panic. Be careful with fear-driven blocks. “AI will steal my traffic” is a reason to think about training, but it rarely justifies blocking search bots. Those bots are the ones that can send visitors back.

The middle path for AI crawlers: block training, allow search

The middle path blocks AI crawlers used for training and allows AI crawlers used for search, so your pages stay citable without feeding future models. Digital Magma suggests this set to clients who are unsure, because it keeps both doors under control and is easy to reverse later.

# Block training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Allow search
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /wp-admin/

That last caption matters. Under RFC 9309, a crawler that finds a group naming it follows that group only. Your general “User-agent: *” rules no longer apply to it.

So repeat admin, cart, login and search-result exclusions inside each named group. Digital Magma sees this missed on most robots.txt files it reviews.

Google-Extended deserves a note of its own. Blocking it does not stop Googlebot, and Google says it is not a ranking signal. It only limits use of your content for training Gemini models and for grounding in Gemini Apps and Vertex AI.

What blocking AI crawlers does not do

Blocking AI crawlers in robots.txt does not delete content already collected, does not force every bot to obey, and does not control user-triggered fetchers. Each owner’s documentation says something narrower than most site owners assume, so read the exact wording before you rely on a block.

Here are the limits, each from the owner’s own text. Read them slowly.

• No retroactive removal. OpenAI says disallowing GPTBot indicates content should not be used in training. It does not say a new block removes content collected earlier.
• User fetchers differ. OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores robots.txt, because a user asked for the page.
• Robots.txt is voluntary. Cloudflare’s robots.txt setting docs say compliance is voluntary and some operators may ignore it.
• IP blocks can backfire. Anthropic warns that blocking its IP addresses may stop its bots reading your robots.txt, so the opt-out may not hold.
• AI Overviews follow different rules. Google says robots.txt for Googlebot, snippet controls and a Search Console setting manage AI Overviews. Google-Extended does not.

Google added that Search Console setting on 31 August 2026. Its Search generative AI control lets a site exclude itself from AI Overviews and AI Mode. Google says it takes effect in a few days.

For anything private, use a login. RFC 9309 says robots.txt rules are not a form of access authorization.

Which CDN and firewall settings block AI crawlers by accident?

AI crawlers are often blocked by CDN bot settings, firewall rules or security plugins rather than robots.txt. These layers act before robots.txt is read, so a friendly file can sit behind a wall that returns errors or challenge pages to every AI search bot.

Cloudflare is the most common example in Digital Magma’s audits. Its Block AI Bots page now sorts bots into Search, Agent and Training groups.

Cloudflare says that from 15 September 2026, new domains block Training and Agent bots on pages that show ads, while Search stays allowed. Mixed-purpose crawlers that do both search and training are blocked by every training block.

That last point is easy to miss. A crawler classed as mixed-purpose loses search access when you block training.

Other accidental blocks Digital Magma has found:

• A WordPress security plugin with a “block bad bots” list that includes AI crawlers
• A firewall rate limit that challenges any client without cookies
• A country block that stops crawlers running from US data centres

Each of these hides well. None of them shows up in the CMS, and none of them edits robots.txt. You only find them by reading the logs or the security dashboard.

Perplexity’s documentation gives a model fix. It suggests a WAF allow rule that combines the user-agent name with its published IP ranges.

What Digital Magma has seen when AI crawlers were blocked or allowed

In Digital Magma’s work, the sites that appear in AI answers are the ones whose crawler access is boring: allowed, tested and logged. The welding manufacturer below never needed a clever trick. It needed open doors for search bots and pages worth citing.

A Canadian welding manufacturer has worked with us for three years. It holds 40+ keywords in Google’s top 10 and is visible in ChatGPT and Google AI Overviews.

In one three-month period its site drew about 30K clicks from about 100K impressions. Search bots could reach every product page, and the content answered the buyer’s questions directly.

The UK tele-oncology startup we rebuilt shows the access idea from another side. Its old React site showed little text until scripts ran.

After the move to Next.js, 30+ keywords ranked on Google in the first month and 8 reached #1. That is a Google outcome. It still makes the point that a crawler can only cite text it receives.

Our clients often value the audit for the same reason. Paramjeet Singh wrote in a Google review: “Their SEO audit was very detailed. We never expected there could be so many technical issues.”

A 15-minute AI crawler check you can run today

A 15-minute AI crawler check covers four places: robots.txt, the CDN, the server logs and a live test. Digital Magma runs it in that order, because each step confirms what the one before it intended. Write down what you find as you go.

1. Minutes 1 to 4: read robots.txt. List every AI token it names. Mark each as training, search or user fetcher. Check that shared exclusions are repeated in each named group.
2. Minutes 5 to 8: open the CDN. Check AI bot policies, custom firewall rules and rate limits. Note anything set to block or challenge.
3. Minutes 9 to 12: search the logs. Filter for OAI-SearchBot, PerplexityBot and Claude-SearchBot. Look at the status codes. A run of 403 errors means a wall.
4. Minutes 13 to 15: test live. Ask ChatGPT and Perplexity a question your site answers, with search on. See whether your pages are cited.

Keep it short. Then go deeper only where a step failed.

Verify any bot by IP as well as name. OpenAI and Perplexity publish JSON lists of their IP ranges, and anyone can fake a user-agent string.

Recheck after any change. OpenAI says search changes take about 24 hours. Perplexity says changes may take up to 24 hours too.

AI crawlers: frequently asked questions

These AI crawler questions come up in almost every robots.txt review Digital Magma runs. The answers stick to what OpenAI, Anthropic, Perplexity, Google and Cloudflare document, and they say so where the documentation is silent. Where an owner is silent, the answer says so.

Related AI crawler questions, sources and author

These pages cover AI crawlers and the topics around them in more depth. Digital Magma marks entries that are still being written, and links live knowledge-base terms to their own dated pages. Begin with the GPTBot entry, because OpenAI documents its bots in the most detail.

• GPTBot: OpenAI’s training crawler and its robots.txt rules
• llms.txt: a separate file some AI tools read, and what it does not control
• AI crawler, Robots.txt for AI and Google-Extended (planned entries)
• Why is my website not in ChatGPT? and Does schema help AI search?

Sources: OpenAI’s Overview of OpenAI Crawlers, Anthropic’s help article on crawling, Perplexity’s crawler documentation, Google’s common crawlers page and Search generative AI control help page, Cloudflare’s Block AI Bots and robots.txt setting docs, and IETF RFC 9309. All were checked on 22 October 2026.