Crawl to Refer Ratio

Crawl to refer ratio is the number of pages a platform's bots crawl on websites for every visitor that platform sends back, a measure of how much an AI or search company takes from the web compared with what it returns.

Also called
Crawl to referral ratio, Crawl to click ratio
MentionBird glossary card with the heading crawl to refer ratio, its one sentence definition, and an A to Z grid of pastel letter tiles with the C tile highlighted
On this page
Key takeaways
  • Crawl to refer ratio compares the pages a platform's bots crawl with the visitors it sends back to websites.
  • Traditional search engines usually sit in single digits, AI search in the hundreds, and training focused crawlers in the thousands or more.
  • Ratios for several AI companies have fallen sharply since 2025 as more AI products search the web and cite sources.
  • Separating search bots from training bots is often the most practical way to act on the ratio.
  • MentionBird shows which of your pages AI platforms cite, the side of the ratio that sends visitors back.

How the ratio works

The crawl-to-refer ratio shows how many pages an AI platform crawls for every visitor it sends to websites. For example, a ratio of 1,000:1 means the platform crawls 1,000 pages for every visitor it sends back.

Cloudflare introduced this metric in 2025, to compare crawling activity and referral traffic across AI and search platforms. It helps website owners understand how much traffic they receive in return for their content.

How ratios compare across platforms

Type of platformTypical ratio in 2026Why
Traditional search enginesSingle digitsResults pages are built to send people to websites
AI search with citationsHundredsAnswers appear on screen, with some clicks through linked sources
Crawlers focused on trainingThousands or morePages read for training send no visitors back on their own

These are broad ranges drawn from Cloudflare Radar data, and the exact figures for each company shift from month to month.

Why the numbers keep moving

Ratios for several AI companies have generally fallen sharply since 2025, in some cases from tens of thousands of pages per visit to the low thousands or hundreds. These changes generally happen when AI platforms adjust their crawling activity or send more visitors to websites through cited links.

Because the figures move so quickly, different reports often show very different numbers for the same company. It is usually more useful to follow the direction of the trend than any single figure.

What the ratio means for your site

The ratio helps decide which bots to allow. Most AI companies now run separate crawlers for training and for search, and treating them differently is often the most practical approach.

Bot typeWhat it generally gives backCommon approach
Search botsCitations and visits when your pages appear in answersUsually worth allowing
Training botsPresence in future models, but no direct visitsAllow or block depending on your priorities
User fetch botsA live read of a page a person asked aboutUsually worth allowing

The visits that do come back show up as AI referral traffic, the other half of the ratio. Before changing any rules, it also helps to confirm which AI crawlers can currently reach your pages, which the free AI crawler accessibility checker shows in seconds.

How MentionBird helps

When AI platforms link to your pages, users can follow those links and visit your website. MentionBird runs your prompts every day across the AI platforms in your plan. It shows where each platform mentions your brand. Starter covers ChatGPT and Gemini, Growth adds Perplexity and Claude, and Custom supports up to 20 platforms: ChatGPT, Gemini, Perplexity, Claude, Grok, Google AI Overviews, Microsoft Copilot, DeepSeek, Google AI Mode, Mistral Vibe, Qwen Chat, Meta AI, Kimi AI, You.com, Ask Brave, Alexa for Shopping, Doubao, ERNIE, Poe, and Duck.ai.

Citation tracking identifies which of your pages AI platforms cite, helping you find opportunities to earn referral traffic. Where your brand is missing, ranking advice suggests improvements, such as creating new content, updating SaaS feature pages, or earning backlinks from frequently cited websites. These changes can help improve your chances of earning AI citations and referral traffic.

FAQs

1. Who measures crawl to refer ratios?

Cloudflare publishes them through Cloudflare Radar, based on traffic across its network, and many reports and studies draw on that data.

2. Is a high ratio always bad?

Not necessarily. A platform may crawl heavily today and send more visitors later as its search features grow, so the trend matters as much as the number.

3. Does blocking a training crawler stop AI from mentioning me?

No. Models can still mention you based on earlier training and other sources, and AI search bots can still cite you if you allow them.

4. Why do reports show different ratios for the same company?

They usually measure different time windows, and ratios have been changing quickly, so a figure from a few months ago may already be out of date.

5. Can I measure crawl to refer ratio for my own site?

Yes, roughly. Compare bot visits per AI platform in your server logs with the referral visits from that platform in your analytics.

See which of your pages AI platforms cite

First month $19 with code TRY19, then $79/mo. Cancel anytime.