AI search glossary
What Is an AI Crawler, and Which Ones Visit Your Site?
Unknown bots in your logs, or an AI-bot switch in Cloudflare, raise the same question. This entry sorts every documented AI crawler by purpose, quotes each owner on what blocking does, and shows how to check the real ones.
What is an AI crawler? The short answer
An AI crawler is a bot run by an AI company that fetches web pages for one of three jobs: collecting text to train models, building an index for AI search answers, or reading a page live because a user asked about it. Each job has its own bot, and robots.txt treats them differently.
Key takeaways
- Training bots, search bots and user-triggered fetchers do different jobs.
- Blocking a training bot does not remove you from that company’s search.
- User-triggered fetchers may not follow robots.txt, by their owners’ own account.
- Check bots against published IP lists, never the user-agent string alone.
- CDN and firewall rules can block AI search bots that robots.txt allows.
AI crawler types: training, search and user-triggered fetchers
AI crawlers fall into three types by purpose. Training crawlers collect pages that may go into future models. Search crawlers index pages so an AI product can cite them in answers. User-triggered fetchers visit one page at the moment a person asks a question that needs it. OpenAI adds a fourth, narrow case for ads.
- Training: GPTBot, ClaudeBot, and Google-Extended as a control token.
- Search: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
- User-triggered: ChatGPT-User, Claude-User, Perplexity-User.
- Ads: OAI-AdsBot, which only visits pages submitted as ChatGPT ads.
The split decides what blocking costs you. Block a training bot and you opt out of future training. Block a search bot and you lose citations in that product.
User-triggered fetchers are the awkward group. They behave more like a browser acting for a person than like a crawler.
AI crawler user agents: the official list by company
AI crawler user agents are documented by each company on its own pages, and those pages are the only reliable source. The table below lists every agent documented by OpenAI, Anthropic, Perplexity and Google for AI use, with what each owner says happens when you block it. Every entry was checked on 22 October 2026.
| Agent | Owner | Type | What blocking does, per the owner |
|---|---|---|---|
| GPTBot/1.4 | OpenAI | Training | Content should not be used to train foundation models |
| OAI-SearchBot/1.4 | OpenAI | Search | Site not shown in ChatGPT search answers |
| ChatGPT-User/1.0 | OpenAI | User-triggered | Robots.txt rules may not apply |
| OAI-AdsBot/1.0 | OpenAI | Ads | Visits only pages submitted as ads |
| ClaudeBot | Anthropic | Training | Future material excluded from training datasets |
| Claude-SearchBot | Anthropic | Search | May reduce visibility in Claude’s search results |
| Claude-User | Anthropic | User-triggered | Claude cannot fetch your pages for user questions |
| PerplexityBot/1.0 | Perplexity | Search | Leaves Perplexity’s search results |
| Perplexity-User/1.0 | Perplexity | User-triggered | Generally ignores robots.txt |
| Google-Extended | Control token | No Gemini training or Gemini Apps grounding. No effect on Search. |
Version numbers change. OpenAI says its example strings may move on, so match the bot name, not the full string.
Google-Extended never appears in logs. Google says crawling uses its existing user agents, and the token only works in robots.txt.
Other bots, such as CCBot or Bytespider, are not in this table. We list only crawlers whose owners document them on a page we could fetch and check.
AI crawler rules in robots.txt: how they apply
AI crawler rules in robots.txt follow the Robots Exclusion Protocol, published by the IETF as RFC 9309. Each crawler looks for the group naming its product token, such as GPTBot, and obeys the rules inside. A crawler with no matching group falls back to the rules for all user agents.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /That file opts out of training at OpenAI and Anthropic while staying open to their search bots and Perplexity’s.
RFC 9309 adds two details worth knowing. The most specific, longest matching rule wins, and crawlers must parse at least 500 kibibytes of the file.
It also says plainly that these rules “are not a form of access authorization”. Robots.txt is a request, not a lock.
Anthropic supports the non-standard Crawl-delay line for ClaudeBot. It asks for a robots.txt file on every subdomain you want covered.
Timing varies. OpenAI says search changes take about 24 hours. Perplexity says up to 24 hours.
AI crawler verification: server logs and published IP lists
AI crawler verification means checking that a request claiming to be GPTBot or PerplexityBot really came from that company. Anyone can fake a user-agent string. The fix is to match the request’s IP address against the IP list the owner publishes, then trust only the requests that match.
- OpenAI: gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json on openai.com.
- Perplexity: perplexitybot.json and perplexity-user.json.
- Anthropic: a published list of source IP addresses, linked from its help article.
OpenAI adds a helpful marker. When its bots fetch robots.txt, the user agent can carry a robots.txt marker, which separates those requests in logs without paths.
Start with a simple filter. Search access logs for the bot names above, then group the hits by IP and compare.
A request with the right name and the wrong IP is an impostor. Treat it like any unknown scraper.
Digital Magma runs this check on a sample of recent logs during an audit. A week of traffic is usually enough to see which real AI bots visit and how often.
AI crawlers, CDNs and firewalls
AI crawlers can be blocked by a CDN or firewall even when robots.txt allows them, and robots.txt will never show it. Security rules, bot-fight modes and rate limits act before the request reaches your site. For AI search visibility, this hidden layer can be where the real block sits.
Cloudflare made this a bigger issue. On 1 July 2025, it said it was changing its default to block AI crawlers unless they pay creators.
Check what your own zone is set to. A default meant for training bots can catch search bots too, if the rule is broad.
Perplexity’s documentation gives sample rules for Cloudflare WAF and AWS WAF. Its advice is to match the user agent and the IP range together.
Anthropic warns against the opposite mistake. Blocking its IPs may not opt you out reliably, because it stops ClaudeBot reading your robots.txt.
AI crawler access for a business: why it matters
AI crawler access matters because it decides, before any content work, whether AI search tools can cite you at all. Block the search bots by accident and ChatGPT, Claude or Perplexity cannot use your pages. Allow training bots without thinking and your content may train future models you never chose to support.
Digital Magma does not push one answer. Neither choice is wrong in itself. A news publisher may block training and keep search open. A manufacturer may allow both.
What hurts is not knowing. Digital Magma treats the crawler map as the first page of any AI visibility audit.
Classic search bots still matter most. Googlebot and Bingbot feed Google’s and Microsoft’s AI answers, and Bing says it respects robots.txt.
AI crawler checks in our client work
AI crawler checks open every Digital Magma technical audit. We map each documented bot against robots.txt, CDN rules and server logs, then agree with the client which ones to allow. The list sits in a short table they can keep and update.
A Canadian welding equipment manufacturer we have worked with for three years appears in ChatGPT and AI Overviews. Open access for the search bots is one of the preconditions for that.
A client review sums up how audits land: “Their SEO audit was very detailed. We never expected there could be so many technical issues.” That was Paramjeet Singh, a Digital Magma client.
AI crawler: frequently asked questions
AI crawler questions from developers and site owners, answered briefly.
What are AI crawlers?
They are bots run by AI companies that fetch web pages to train models, index pages for AI search, or read a page live for a user’s question.
How do I check whether AI crawlers can access my site?
Read your robots.txt for each bot name, check CDN and firewall rules, then search server logs for the bots and match their IPs to the published lists.
Do AI crawlers follow robots.txt?
The documented training and search bots say they do. User-triggered fetchers such as Perplexity-User and ChatGPT-User may not, according to their owners.
Which AI crawlers are used for training?
GPTBot for OpenAI and ClaudeBot for Anthropic. Google uses the Google-Extended token to control training of Gemini models. Perplexity says its bots are not used for that.
Should I block AI crawlers?
Decide per type. Keep search bots open if you want AI citations. Training bots are a business choice. Blocking user fetchers stops assistants reading pages customers ask about.
Does Cloudflare block AI crawlers by default?
Cloudflare said on 1 July 2025 it was changing its default to block AI crawlers unless they pay creators. Check your own dashboard settings, since zones differ.
How do I find AI crawlers in my server logs?
Search the access log for names like GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, then confirm each IP against the owner’s published list.
AI crawler access, mapped and fixed
Digital Magma maps every AI crawler against your robots.txt, CDN and logs, and agrees with you which to allow. Start with a free audit of your robots.txt and AI crawler access. See our AI SEO service or book a free SEO audit.
No AI citation is guaranteed. Call +91 79823 19697, email contact@digitalmagma.com or message on WhatsApp (Mon–Sat, 9 am–9 pm IST). You can read our Google reviews before you get in touch.