Check robots.txt rules and a live fetch for 14 AI crawlers, from GPTBot to PerplexityBot. The live test catches the Cloudflare and firewall blocks that robots.txt-only checkers miss.
Content quality, schema markup, a citations plan: all of it assumes a crawler can fetch the page. If it can’t, the rest is theoretical. When a buyer asks ChatGPT for the best option in your category, the answer gets written with whoever is reachable, and nothing shows up in your analytics.
An AI crawler access check tests whether the bots that AI platforms send across the web can actually retrieve a given page. It compares what a site says it permits with what its servers really return to each bot. The result is a verdict for every crawler, showing which AI products can use the page as a source, which cannot, and what is stopping them.
It’s the most common cause of a block that robots.txt can’t explain. CDN and firewall bot rules act before robots.txt is read: they refuse a request by user agent, often with a 403 or a challenge page, so a site can publish an open robots.txt and still turn every AI crawler away.
That’s why this checker sends a live request as each bot and compares it with a normal browser request. If robots.txt says allowed but the live request is refused, look at your CDN’s bot management or AI crawler settings and its security event log.
One limit worth knowing: our requests use each crawler’s user agent, not its IP address. If your firewall verifies bots by IP range, the real crawler can be treated differently. Your server logs are the final word.
The checker groups all 14 in the order that affects you soonest. Search and browsing crawlers decide whether you can appear in an answer today. Training crawlers decide what a future model knows about you. Teams that blocked “AI bots” a year ago usually blocked the third group and left the first two open — or did the reverse without meaning to.
These build the index an AI answer is drawn from. Block them and you aren't a candidate for the answer at all — the answer still gets written, using whoever is reachable.
These fetch a page live, in the moment, because someone asked a question the model needs to look up. A block here surfaces to the user as “I couldn't access that page.”
These feed the next model. Blocking them is a legitimate choice and plenty of publishers make it deliberately — but it changes nothing about the answer someone gets today.
Once crawlers can reach a page, check whether AI would quote it with the GEO Score Checker.
Two starting points for the robots.txt file at the root of your domain. Keep the rules you already have, add one of these, then run the checker again after 24 hours.
First month $19 with code TRY19, then $79/mo. Cancel anytime.