Common Crawl
Common Crawl is a nonprofit organization that regularly crawls the public web and publishes the data for free, creating one of the largest open archives of web pages, which many AI models have used as training data.
- Common Crawl is a nonprofit that regularly crawls the public web and releases the data free for anyone to use.
- Filtered Common Crawl data made up about 60% of the training mix for GPT 3, and many later models have drawn on it too.
- The archive captures public, crawlable page text as it looked at crawl time, and usually misses gated, blocked, or script loaded content.
- Allowing CCBot keeps your pages in future archives, while blocking it limits reuse but does not remove earlier snapshots.
- MentionBird shows how AI describes your brand, including in answers drawn from training data, and what to fix where those answers fall short.
What Common Crawl is
Common Crawl is a nonprofit founded in 2007 that regularly crawls the public web and releases what it collects for anyone to use. Each crawl captures billions of pages, and the archive now spans many years of the web.
Researchers, universities, and companies use it because building a crawl of that size from scratch is expensive. Its crawler is called CCBot, one of several AI crawlers that site owners can allow or block.
Why it matters for AI
Common Crawl has been one of the most widely used sources of training data for large language models. In the research paper behind GPT 3, OpenAI listed a filtered version of Common Crawl as about 60% of the model's training mix, and many later models have drawn on it in some form.
That gives it a lasting influence on what models know. When an assistant answers from memory rather than fetching live pages, as it often does outside AI search, it is drawing partly on the version of the web that archives like Common Crawl captured.
What ends up in the archive
| Usually included | Usually missing |
|---|---|
| Public pages the crawler can reach | Pages behind logins or paywalls |
| Pages that allow CCBot | Pages blocked for CCBot in robots.txt |
| Text in the page HTML | Content that only appears after scripts run |
| The page as it looked at crawl time | Any change made after that crawl |
Should you allow or block CCBot?
There is no single right answer. It depends on whether your priority is presence in future AI models or control over how your content is reused.
| Choice | What it generally means | Who it tends to suit |
|---|---|---|
| Allow CCBot | Your public pages can appear in future archives and in models trained on them | Brands that want AI to know about their products and positioning |
| Block CCBot | Future crawls skip your site, though earlier archives usually keep what they already captured | Publishers and creators who want to limit reuse of their content |
Blocking takes a single robots.txt rule naming CCBot, and it does not affect search engines that use their own crawlers.
What it means for your brand
- Training data lags behind your site: A model can carry an older picture of your brand long after you have updated your pages.
- Old pages can live on: Retired pricing or outdated descriptions captured in past crawls may still inform answers from memory.
- Clear HTML travels further: Key facts written in the page text are more likely to be captured than details that depend on scripts.
- Accuracy now pays off later: Every crawl is a fresh snapshot, so keeping public pages accurate improves what future archives record.
Other glossary terms to check out
FAQs
1. Is Common Crawl the same as Google's crawler?
No. Common Crawl is an independent nonprofit, and its archive is separate from Google's search index.
2. Can I remove my site from past crawls?
Generally not. Blocking CCBot stops future crawls, but pages captured in earlier archives typically stay in them.
3. Does blocking CCBot affect my Google rankings?
No. Google uses its own crawler, so blocking CCBot does not change how Google indexes or ranks your pages.
4. Do all AI companies use Common Crawl?
Many have used it, but most large AI companies also run their own crawlers and combine several data sources.
5. How often does Common Crawl collect new data?
It publishes new crawls regularly, usually every month or two, each capturing a fresh snapshot of billions of pages.
See what AI says about your brand today
First month $19 with code TRY19, then $79/mo. Cancel anytime.
.png)
