✦ Free Tool  ·  No Login Required

Free AI Crawler Checker

Can ChatGPT, Claude and Perplexity reach your site?

Check robots.txt rules and a live fetch for 14 AI crawlers, from GPTBot to PerplexityBot. The live test catches the Cloudflare and firewall blocks that robots.txt-only checkers miss.

🤖 14 AI crawlers  ·  🔍 robots.txt + live fetch checks  ·  🔗 Shareable results
The one check everything else assumes

What is an AI crawler access check?

Content quality, schema markup, a citations plan: all of it assumes a crawler can fetch the page. If it can’t, the rest is theoretical. When a buyer asks ChatGPT for the best option in your category, the answer gets written with whoever is reachable, and nothing shows up in your analytics.

DEFINITION

An AI crawler access check tests whether the bots that AI platforms send across the web can actually retrieve a given page. It compares what a site says it permits with what its servers really return to each bot. The result is a verdict for every crawler, showing which AI products can use the page as a source, which cannot, and what is stopping them.

robots.txt is half the story

Is Cloudflare blocking AI crawlers from your site?

It’s the most common cause of a block that robots.txt can’t explain. CDN and firewall bot rules act before robots.txt is read: they refuse a request by user agent, often with a 403 or a challenge page, so a site can publish an open robots.txt and still turn every AI crawler away.

That’s why this checker sends a live request as each bot and compares it with a normal browser request. If robots.txt says allowed but the live request is refused, look at your CDN’s bot management or AI crawler settings and its security event log.

One limit worth knowing: our requests use each crawler’s user agent, not its IP address. If your firewall verifies bots by IP range, the real crawler can be treated differently. Your server logs are the final word.

Three jobs, not one

Which AI crawlers does the checker test?

The checker groups all 14 in the order that affects you soonest. Search and browsing crawlers decide whether you can appear in an answer today. Training crawlers decide what a future model knows about you. Teams that blocked “AI bots” a year ago usually blocked the third group and left the first two open — or did the reverse without meaning to.

1

AI search crawlers

These build the index an AI answer is drawn from. Block them and you aren't a candidate for the answer at all — the answer still gets written, using whoever is reachable.

✓
OAI-SearchBot — ChatGPT search
✓
Claude-SearchBot — Claude search
✓
PerplexityBot — Perplexity's index
2

AI assistant browsers

These fetch a page live, in the moment, because someone asked a question the model needs to look up. A block here surfaces to the user as “I couldn't access that page.”

✓
ChatGPT-User — live browsing
✓
Claude-User — live browsing
✓
Perplexity-User — live browsing
3

AI training crawlers

These feed the next model. Blocking them is a legitimate choice and plenty of publishers make it deliberately — but it changes nothing about the answer someone gets today.

✓
GPTBot · ClaudeBot · Google-Extended
✓
Applebot-Extended · Meta-ExternalAgent · Amazonbot
✓
Bytespider · CCBot
Check
robots.txt-only checkers
MentionBird
Reads robots.txt rules for each AI bot
Yes
Yes
Sends a live request with each bot’s user agent
No
Yes, all 14 crawlers
Sends a control request as a normal browser
No
Yes
Shows the Disallow line behind a block
Sometimes
Yes, for blocked crawlers
Reports whether llms.txt exists
Sometimes
Yes
Login needed
Usually not
No
When a row comes back blocked

What to do when a crawler is blocked

Once crawlers can reach a page, check whether AI would quote it with the GEO Score Checker.

Blocked by robots.txt
Delete or narrow the Disallow line shown in that crawler’s row, for the search and browsing crawlers you want in AI answers.
Allowed by robots.txt, refused live
Change the CDN, firewall or security plugin rule that blocks that user agent.
Check again after 24 hours
Results are cached for up to 24 hours per URL and crawler, so an immediate re-check may still show the old verdict.
Copy, paste, re-check

robots.txt rules for AI crawlers

Two starting points for the robots.txt file at the root of your domain. Keep the rules you already have, add one of these, then run the checker again after 24 hours.

Allow every AI crawler

For the widest AI visibility. Search, live browsing and training crawlers can all read your site.

# AI search, live browsing and training crawlers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: CCBot
Allow: /

Allow AI answers, block AI training

Stay eligible for ChatGPT, Claude and Perplexity answers today while opting out of model training.

# AI search and live browsing: allowed (these put you in answers)
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# AI training: blocked (no effect on today's answers)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: CCBot
Disallow: /

Before you paste: a crawler follows only the most specific group that names it. If your current robots.txt has a User-agent: * group with Disallow lines you want AI bots to respect too, such as /admin/ or /cart/, copy those lines into these groups. And robots.txt can't override a firewall: if the checker shows a crawler allowed by robots.txt but refused live, change the CDN or bot-protection rule instead.

AI crawler access checker FAQ

Does blocking AI crawlers actually cost me customers?

It costs you the chance to be in the answer. When someone asks ChatGPT or Perplexity for the best option in your category, the answer is assembled from pages the platform can fetch and cite. If your domain isn't reachable by the search and browsing crawlers, you're not a candidate — the answer still gets written, using whoever is reachable.

You won't find this in your analytics, because there's no visit to measure and nothing to attribute. It surfaces as competitors being named instead of you. The one place it does leave a trace is your server logs: blocked fetch attempts from AI user agents show up there even when your analytics shows nothing at all.

If every crawler can reach my site, will AI cite me?

Not automatically. Access is the precondition, not the outcome. Once the search and browsing crawlers can fetch your pages, whether ChatGPT or Perplexity names you depends on what those pages say and who else covers the topic. The free AI visibility report shows whether ChatGPT names your brand today.

Which 14 crawlers do you check?

Four groups, because they don't all do the same job.

AI search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot. These build the indexes that ChatGPT search, Claude, and Perplexity draw on when they answer.

Live-browsing agents — ChatGPT-User, Claude-User, Perplexity-User. These fetch a page in the moment, when someone's question needs looking up. Worth knowing: Perplexity documents Perplexity-User as user-triggered and not bound by robots.txt, so a disallow rule won't necessarily stop it.

Training crawlers — GPTBot, ClaudeBot, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot. These collect content for model training. They send no visitors and generate no citations.

Opt-out tokens — Google-Extended and Applebot-Extended. These aren't crawlers at all. Nothing fetches with those names, and you'll never see them in a server log. They're permission flags you set in robots.txt to control whether Google and Apple may use content their regular crawlers already collected for AI training.

The first six decide whether you can appear in an answer today. The last eight decide what future models know about you.

Why check twice, with robots.txt and a live fetch?

Because the two can disagree, and the disagreement is the useful part.

robots.txt is a request a crawler honors voluntarily. A CDN or WAF bot rule is enforcement — it rejects the request by user agent before robots.txt is ever read. A site can publish a completely permissive robots.txt and still return 403 to every AI crawler that tries. A check that read only the file would report that site as fully open.

So every crawler gets both: its robots.txt rules evaluated against the URL you entered, and a live fetch of that URL sent with its user agent.

Two of the fourteen work a little differently. Google-Extended and Applebot-Extended are permission tokens, not crawlers, so nothing really fetches with those names. For those two, the robots.txt result is the one that reflects reality — the live fetch is a supplementary probe for the uncommon case where a firewall rule keys on the token string itself.

Can it tell me if Cloudflare is blocking AI crawlers?

It shows the symptom. If robots.txt allows a crawler but the live request comes back 403 or as a challenge page, something at the network layer is refusing that user agent, and on many sites that is a Cloudflare or other CDN bot rule. The control request with a normal browser user agent tells you whether the block targets AI bots specifically. To confirm the cause, check your CDN’s bot or AI crawler settings and its security event log.

How do I check if GPTBot is blocked on my site?

Enter your URL above. The GPTBot row shows two results: what your robots.txt says about GPTBot, and what your server returned to a live request sent with GPTBot’s user agent. If the robots.txt file blocks it, the row shows the Disallow line that decided it. Blocking GPTBot only affects training; ChatGPT search uses OAI-SearchBot and ChatGPT-User.

If I block GPTBot, am I out of ChatGPT?

No, and this is the most common misreading of AI crawler rules.

GPTBot collects data for model training. OAI-SearchBot builds the index ChatGPT search draws from. ChatGPT-User fetches pages live when a user asks something that needs looking up. Blocking GPTBot leaves both of the others untouched.

The mistake runs both ways. Teams who blocked training often believe they've opted out of AI search, and teams who allowed search often believe they've allowed training. They're separate permissions with separate consequences, which is why this check reports them separately.

How is this different from Google Search Console’s robots.txt report?

Search Console reports how Google’s own crawlers read your robots.txt. It doesn’t test OAI-SearchBot, ClaudeBot, PerplexityBot or the other AI user agents, and it doesn’t send a live request as each one, so it can’t show a firewall or CDN rule that turns AI bots away. Use both: Search Console for Googlebot, this checker for the 14 AI crawlers.

Should I allow the training crawlers?

That's a business decision, not a technical one, and there's a real argument on both sides.

Blocking them keeps your content out of models you don't control. Allowing them means future models carry some baseline knowledge of your brand without having to look you up.

What's worth separating is the timeline. Training access affects models released later — quarters or years from now. Search and browsing access affects the answer someone gets this afternoon. If you only change one thing, change the second.

What does "Partial access" mean?

Either the two checks disagreed, or robots.txt allows the crawler on the URL you checked while disallowing it elsewhere on your site.

The row tells you which. You'll see a "Partial access" verdict next to a "robots.txt: Partial" pill, and underneath, a line stating how many other path rules are disallowed for that crawler. If a specific rule matched the URL you entered, that rule is shown too.

Note that you get the count, not the list. So you'll know other paths are restricted without seeing which — you'll need to read your robots.txt for that. The shape to watch for: a crawler that can reach your homepage but is disallowed from the directory holding your product or documentation pages, which is usually the exact content you most want quoted. Rows that come back fully blocked do show the specific Disallow line that decided it.

What if the checker can't reach my site at all?

You get "Could not reach this site" rather than a score of zero, because those are different problems.

Every scan includes a control fetch using an ordinary browser user agent. If your site refuses that too — a 401, a 403, a challenge page — then the refusal isn't about AI crawlers specifically, and reporting it as fourteen blocks would be misleading. Each crawler comes back as unknown rather than blocked, with a note that the site refused a normal browser as well.

That's still worth knowing, because aggressive bot management refuses real crawlers the same way it refuses ours. But it isn't the same finding as a site that deliberately names and blocks fourteen user agents.

One exception: if your robots.txt explicitly disallows some of the fourteen, those rows still report as blocked, since robots.txt needs no live fetch to evaluate. You'll see a mixed result in that case rather than the unreachable message.

Why does the scan take up to a minute?

Requests are paced deliberately. We wait at least a second between fetches to your site, and a full scan makes around seventeen of them: robots.txt, llms.txt, one control fetch with a normal browser user agent, and one per crawler. Firing those simultaneously would look exactly like the traffic pattern bot-management rules exist to stop, and could get the checker itself blocked — which would make your result meaningless.

Most scans finish comfortably inside a minute. We allow up to one for slow sites, and if a crawler's fetch doesn't complete in time it reports as not checkable rather than blocked. Rate limiting works the same way: a 429 gets one retry, then reports as not checkable. We'd rather tell you we couldn't get an answer than tell you a crawler is blocked when it isn't.

Do you store the URL I check?

We don't store your pages or your robots.txt file. We do log the URL you submit and your IP address, for abuse prevention and rate limiting, and those logs are deleted after 30 days.

Results are cached for up to 24 hours per URL and crawler, so a repeat check will often return without touching your site again. That cache isn't guaranteed — it lives in memory, and a repeat check may run fresh instead. The practical consequence: if you've just unblocked a crawler, a re-check might still show the previous verdict, so wait 24 hours if you want certainty.

MentionBird doesn't use anything you submit to train models.

Crawlers can read you. Now find out whether AI recommends you.

First month $19 with code TRY19, then $79/mo. Cancel anytime.