AI Crawler

An AI crawler is a bot that reads web pages on behalf of an AI company, to train its models, build its AI search index, or fetch a page live when a user asks.

Also called
AI bot, LLM crawler, AI web crawler
MentionBird glossary card with the heading AI crawler, its one sentence definition, and an A to Z grid of pastel letter tiles with the A tile highlighted
On this page
Key takeaways
  • AI crawlers visit your pages for three main reasons: to collect training data, to build AI search indexes, or to fetch a page when a user asks about it.
  • Each AI company runs separate crawlers for each job, so you can block training without leaving AI search.
  • Blocking search or user bots removes your pages from AI answers, which is rarely what a brand wants.
  • Firewalls, JavaScript heavy pages, and old robots.txt rules often block AI crawlers by accident.
  • Once crawlers can reach you, MentionBird shows whether AI actually cites and mentions your pages.

The three types of AI crawlers

AI companies no longer send one bot to do everything. Most run separate crawlers for separate jobs, and each one can be allowed or blocked on its own.

Three AI crawler jobs: training bots like GPTBot, search bots like OAI-SearchBot, and live fetch bots like ChatGPT-User

The most common AI crawlers you will see in your server logs:

CompanyCrawlerWhat it does
OpenAIGPTBotCollects pages that may be used to train OpenAI models
OpenAIOAI-SearchBotIndexes pages so they can appear in ChatGPT search results
OpenAIChatGPT-UserVisits a page when a ChatGPT user or action asks for it
AnthropicClaudeBotCollects pages that may be used to train Claude models
AnthropicClaude-SearchBot and Claude-UserIndex pages for Claude search and fetch pages users ask about
PerplexityPerplexityBot and Perplexity-UserIndex pages for Perplexity answers and fetch pages live
GoogleGoogle-ExtendedA robots.txt setting that controls whether Google may use your pages for Gemini training
Common CrawlCCBotBuilds an open web archive that many AI models train on

AI crawlers vs Googlebot

Googlebot crawls your site so Google can rank it. AI crawlers crawl it so a model can learn from it or quote it. The mechanics look similar, but three differences matter.

  • They skip what they cannot read easily: Many AI crawlers do not run JavaScript the way Googlebot does, so content that only appears after scripts load may be invisible to them.
  • They are split by purpose: Blocking Googlebot removes you from Google. Blocking an AI training bot does not have to remove you from AI search, as long as the search bot is still allowed.
  • They are newer and easy to block by mistake: Security tools and hosting settings often treat unfamiliar bots as threats, so AI crawlers get blocked while Googlebot sails through.

Should you block AI crawlers?

It depends on which ones. Blocking training bots is a reasonable choice if you do not want your content used to train models. Blocking search and live fetch bots is almost never what a brand wants, because it takes your pages out of the answers buyers read.

Two robots.txt examples: blocking OAI-SearchBot removes a site from ChatGPT search, while blocking only GPTBot keeps it in

If AI search visibility matters to you, the safest default is to allow the search and user bots from every platform your buyers use. A page AI cannot read can never become an AI citation.

Why AI crawlers get blocked by accident

  • A blanket robots.txt rule: A rule that blocks every bot, often left over from a staging site, shuts out AI crawlers along with everything else.
  • Firewall and bot protection: Some CDN and security settings block AI bots by default, even when robots.txt allows them.
  • Content that needs JavaScript: Prices, features, or reviews that load through scripts can look empty to a crawler that does not run them.
  • Logins and pop ups: Pages behind sign ins, cookie walls, or region gates return nothing useful to a bot.
  • Old rules copied from templates: Robots.txt files often carry rules written years ago for bots and paths that no longer exist.

How to check if AI crawlers can reach your site

  1. Read your robots.txt: Open yoursite.com/robots.txt and look for rules that name AI crawlers or block everything.
  2. Check your CDN and firewall: Look for bot blocking or AI crawler settings that override robots.txt.
  3. Test a key page without scripts: Load your pricing or product page with JavaScript turned off and see what text is left.
  4. Look at your server logs: Search for AI crawler names to see which ones visit and which get errors.

Want the quick version? Our free AI crawler accessibility checker tests your site against the main AI bots in seconds.

Check your site free

How MentionBird helps once crawlers reach your site

Crawler access is the entry ticket, not the result. Once AI can read your pages, MentionBird shows whether AI platforms actually use them. Citation tracking records which of your pages each platform cites, and prompt tracking shows whether those citations turn into mentions across the AI platforms in your plan. Starter covers ChatGPT and Gemini, Growth adds Perplexity and Claude, and Custom supports up to 20 platforms: ChatGPT, Gemini, Perplexity, Claude, Grok, Google AI Overviews, Microsoft Copilot, DeepSeek, Google AI Mode, Mistral Vibe, Qwen Chat, Meta AI, Kimi AI, You.com, Ask Brave, Alexa for Shopping, Doubao, ERNIE, Poe, and Duck.ai.

If a page still never gets cited after you open access, ranking advice points to what else is missing.

FAQs

1. Does blocking GPTBot remove me from ChatGPT?

Not on its own. GPTBot handles training. ChatGPT search relies on OAI-SearchBot, and live visits use ChatGPT-User. You can block GPTBot and still appear in ChatGPT search answers if the other two are allowed.

2. Do AI crawlers follow robots.txt?

The major AI companies say their crawlers follow robots.txt rules. Robots.txt is a request, not a lock, so firewall rules are the only way to enforce a block.

3. Can AI crawlers read JavaScript?

Many cannot, or only partly. Put key facts like prices, features, and product names in the page HTML so every crawler can read them.

4. Will allowing AI crawlers slow down my site?

For most sites, no. AI crawlers make far fewer requests than people often expect. If one bot causes load problems, you can limit it without blocking the rest.

5. How often do AI crawlers visit?

It varies by site and by bot. Popular, frequently updated pages tend to be visited more often. Your server logs show exactly how often each crawler comes by.

See which of your pages AI actually cites

First month $19 with code TRY19, then $79/mo. Cancel anytime.