What Is ClaudeBot, and How Do You Control Anthropic’s Bots?

ClaudeBot and Anthropic’s other two bots: Claude-SearchBot and Claude-User

ClaudeBot is one of three robots that Anthropic’s help article describes, and each one maps to a different product need. Treating them as a single “Claude crawler” is the root of most bad decisions, because blocking the wrong one costs search visibility rather than training exposure.

The pattern matches what OpenAI and Perplexity do. Training, search and user-triggered fetching sit in separate groups. You answer each question on its own.

Short version? Block ClaudeBot if training worries you. Leave the other two alone if Claude search matters. Digital Magma starts every Anthropic review from that split, because it keeps a privacy decision from turning into a visibility loss.

Anthropic’s article is the place to recheck these descriptions. It has a subscribe form for notice of substantial changes.

ClaudeBot collection: what blocking it changes

ClaudeBot collects public web content that, in Anthropic’s words, “could potentially contribute” to training its generative models. Restricting it is a forward-looking signal. It tells Anthropic to leave your future materials out of training datasets. It says nothing about search, and nothing about pages a user asks Claude to read.

That forward-looking wording matters. Anthropic’s article talks about “future materials”, so it does not promise to remove anything gathered before the block.

Blocking ClaudeBot also leaves Claude-SearchBot untouched. A site can refuse training and still be indexed for Claude’s search answers.

Anthropic lists four collection principles in the same article. It aims to be transparent, to avoid disruptive crawl rates, to honour robots.txt, and not to bypass CAPTCHAs.

That last point is useful. A login or CAPTCHA wall is respected.

So when a client asks Digital Magma how to keep a members area away from AI crawlers, the honest answer is a login, with robots.txt as a polite extra.

ClaudeBot robots.txt rules, Crawl-delay and subdomains

ClaudeBot follows robots.txt, and Anthropic says opting out “requires modifying the robots.txt file”. It also supports Crawl-delay, a non-standard line that slows requests instead of stopping them. Pick the block that fits your decision, and place the file in the top-level directory of every host.

# 1. Slow ClaudeBot down, keep it allowed
User-agent: ClaudeBot
Crawl-delay: 1

# 2. No training, stay in Claude search
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

# 3. Keep one folder out of every Anthropic bot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /drafts/

Crawl-delay has a catch. Google’s robots.txt specification says Google does not support it. The line slows ClaudeBot. Googlebot ignores it.

Subdomains need separate files. Anthropic asks owners to add the block “for every subdomain” they want to opt out, so blog.example.com needs its own robots.txt.

Shared exclusions need copying too. Under RFC 9309, a crawler that matches a named group ignores the * group.

Digital Magma repeats admin and login exclusions inside every named group for that reason. Our robots.txt shows the pattern for all three Anthropic bots.

ClaudeBot verification by IP as well as user agent

ClaudeBot verification needs the source address as well as the name. Anthropic publishes its crawler IP list at claude.com/crawling/bots.json and says a request from an address on that list “is coming from Anthropic”. Any request claiming the name from elsewhere is someone borrowing it.

Anthropic’s help text does not print a full user-agent string. So match on the token “ClaudeBot” in your logs, then test each address:

grep -i "claudebot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn

Compare those addresses with Anthropic’s bots.json. The copy we fetched on 26 October 2026 listed 14 IPv4 prefixes, created on 20 October 2026.

The list is short and recently dated, which suggests it changes. Fetch it fresh before each review instead of pasting ranges into a firewall once.

Heavy crawl volume from a listed address is a Crawl-delay problem. Heavy volume from unlisted addresses is a scraper problem, and a firewall rule is the right tool.

ClaudeBot decisions for a business: why they matter

ClaudeBot decisions are really two business questions wearing one technical name. Do you want future Claude models to learn from your public pages? And do you want Claude’s search answers to cite you? Digital Magma asks clients to answer both in writing before anyone touches robots.txt.

A publisher selling its archive may say no to training. A consultancy that wants Claude to describe its services accurately may say yes.

Search is a different matter. Turning off Claude-SearchBot or Claude-User can cut visibility for people asking Claude about your field, as Anthropic’s own article notes.

Digital Magma allows all three Anthropic bots on its own site. That reflects a firm that wants its expertise widely known, and it is a choice, not a rule.

The same split applies across vendors. Our AI crawler entry maps the training and search bots of each major AI company.

ClaudeBot in our client work

ClaudeBot rarely causes trouble for Digital Magma’s clients, but a blanket block often does. For a Canadian welding equipment manufacturer that appears in ChatGPT and Google’s AI Overviews, keeping every assistant’s search bot open protects visibility built over a three-year engagement.

The welding client’s pages answer precise technical questions. Those are exactly the questions engineers type into Claude, so search access matters more than the training debate.

For a Toronto ENT clinic, the concern is patient trust. Accurate service and doctor details should be reachable by every assistant a patient might ask.

For an app development company with buyers in six countries, subdomains and country folders are the risk. Each host needs a correct robots.txt of its own.

ClaudeBot mistakes: IP blocking and blocking search by accident

ClaudeBot mistakes tend to fall into two groups. Owners either block too bluntly, by IP or with a single rule for every Anthropic name, or they write a correct rule in the wrong place. Each mistake below comes from a point Anthropic makes in its own help article.

• Blocking IPs instead of using robots.txt. Anthropic warns this “may not work correctly or persistently guarantee an opt-out”, because it stops the bot reading your robots.txt.
• Blocking search with training. A rule covering ClaudeBot, Claude-SearchBot and Claude-User together removes you from Claude’s search answers too.
• Forgetting subdomains. A rule on www misses shop and blog. Each host is separate.
• Expecting Crawl-delay to calm Google. Google ignores the line.
• Trusting the name alone. Anyone can copy the ClaudeBot name. Check addresses against bots.json.
• Treating robots.txt as security. RFC 9309 says its rules “are not a form of access authorization”.

ClaudeBot: frequently asked questions

ClaudeBot questions we get from site owners and developers.

ClaudeBot: related terms, sources and author

ClaudeBot links to these Digital Magma entries:

• AI crawler: training, search and user bots
• Robots.txt for AI bots: four rule sets to copy
• GPTBot: OpenAI’s training crawler
• LLM optimization: how models come to describe a brand
• Should you block or allow AI crawlers?

Sources: Anthropic’s help article on crawling (dated 7 April 2026) and its bots.json list, Google Search Central on robots.txt, and IETF RFC 9309. All checked on 26 October 2026.