Crawl to Refer Ratio
Crawl to refer ratio is the number of pages a platform's bots crawl on websites for every visitor that platform sends back, a measure of how much an AI or search company takes from the web compared with what it returns.
- Crawl to refer ratio compares the pages a platform's bots crawl with the visitors it sends back to websites.
- Traditional search engines usually sit in single digits, AI search in the hundreds, and training focused crawlers in the thousands or more.
- Ratios for several AI companies have fallen sharply since 2025 as more AI products search the web and cite sources.
- Separating search bots from training bots is often the most practical way to act on the ratio.
- MentionBird shows which of your pages AI platforms cite, the side of the ratio that sends visitors back.
How the ratio works
The crawl-to-refer ratio shows how many pages an AI platform crawls for every visitor it sends to websites. For example, a ratio of 1,000:1 means the platform crawls 1,000 pages for every visitor it sends back.
Cloudflare introduced this metric in 2025, to compare crawling activity and referral traffic across AI and search platforms. It helps website owners understand how much traffic they receive in return for their content.
How ratios compare across platforms
| Type of platform | Typical ratio in 2026 | Why |
|---|---|---|
| Traditional search engines | Single digits | Results pages are built to send people to websites |
| AI search with citations | Hundreds | Answers appear on screen, with some clicks through linked sources |
| Crawlers focused on training | Thousands or more | Pages read for training send no visitors back on their own |
These are broad ranges drawn from Cloudflare Radar data, and the exact figures for each company shift from month to month.
Why the numbers keep moving
Ratios for several AI companies have generally fallen sharply since 2025, in some cases from tens of thousands of pages per visit to the low thousands or hundreds. These changes generally happen when AI platforms adjust their crawling activity or send more visitors to websites through cited links.
Because the figures move so quickly, different reports often show very different numbers for the same company. It is usually more useful to follow the direction of the trend than any single figure.
What the ratio means for your site
The ratio helps decide which bots to allow. Most AI companies now run separate crawlers for training and for search, and treating them differently is often the most practical approach.
| Bot type | What it generally gives back | Common approach |
|---|---|---|
| Search bots | Citations and visits when your pages appear in answers | Usually worth allowing |
| Training bots | Presence in future models, but no direct visits | Allow or block depending on your priorities |
| User fetch bots | A live read of a page a person asked about | Usually worth allowing |
The visits that do come back show up as AI referral traffic, the other half of the ratio. Before changing any rules, it also helps to confirm which AI crawlers can currently reach your pages, which the free AI crawler accessibility checker shows in seconds.
How MentionBird helps
When AI platforms link to your pages, users can follow those links and visit your website. MentionBird runs your prompts every day across the AI platforms in your plan. It shows where each platform mentions your brand. Starter covers ChatGPT and Gemini, Growth adds Perplexity and Claude, and Custom supports up to 20 platforms: ChatGPT, Gemini, Perplexity, Claude, Grok, Google AI Overviews, Microsoft Copilot, DeepSeek, Google AI Mode, Mistral Vibe, Qwen Chat, Meta AI, Kimi AI, You.com, Ask Brave, Alexa for Shopping, Doubao, ERNIE, Poe, and Duck.ai.
Citation tracking identifies which of your pages AI platforms cite, helping you find opportunities to earn referral traffic. Where your brand is missing, ranking advice suggests improvements, such as creating new content, updating SaaS feature pages, or earning backlinks from frequently cited websites. These changes can help improve your chances of earning AI citations and referral traffic.
FAQs
1. Who measures crawl to refer ratios?
Cloudflare publishes them through Cloudflare Radar, based on traffic across its network, and many reports and studies draw on that data.
2. Is a high ratio always bad?
Not necessarily. A platform may crawl heavily today and send more visitors later as its search features grow, so the trend matters as much as the number.
3. Does blocking a training crawler stop AI from mentioning me?
No. Models can still mention you based on earlier training and other sources, and AI search bots can still cite you if you allow them.
4. Why do reports show different ratios for the same company?
They usually measure different time windows, and ratios have been changing quickly, so a figure from a few months ago may already be out of date.
5. Can I measure crawl to refer ratio for my own site?
Yes, roughly. Compare bot visits per AI platform in your server logs with the referral visits from that platform in your analytics.
See which of your pages AI platforms cite
First month $19 with code TRY19, then $79/mo. Cancel anytime.
.png)
