Content Signals
Content Signals are robots.txt directives, introduced by Cloudflare in 2025, that let a website state separately whether its content may be used for search, for live AI answers, or for training AI models.
- Content Signals extend robots.txt so a site can state how its content may be used, not just which crawlers may visit.
- The three signals cover search, ai-input for live AI answers, and ai-train for model training.
- Cloudflare added search=yes and ai-train=no by default to more than 3.8 million domains in 2025, leaving ai-input unset.
- Compliance is voluntary, so the signals work best alongside crawler level rules for specific bots.
- Brands that want AI citations usually keep search and ai-input open while choosing separately whether to allow training.
Why Content Signals exist
Classic robots.txt rules can only say which crawlers may visit which pages. They cannot say what happens to the content afterward. As AI companies began using the same pages for search results, live AI answers, and model training, site owners needed a way to say yes to one use and no to another.
Content Signals fill that gap. Instead of naming crawlers one by one, a site states its preferences for each use directly, and crawlers that respect the policy are expected to follow them.
The three signals

- search: Permission to build a search index and show links and short excerpts. Cloudflare's definition excludes AI written summaries.
- ai-input: Permission to feed content into AI models at answer time, such as when an assistant reads a page to answer a question.
- ai-train: Permission to use content to train or fine tune AI models.
Cloudflare's default setting
When Cloudflare introduced Content Signals in September 2025, it added them automatically to more than 3.8 million domains using its managed robots.txt, setting search=yes and ai-train=no. It deliberately left ai-input unset, leaving that decision to each site owner.
Do AI companies follow them?
Content Signals are a stated preference, not a technical block or a legal contract, and compliance is voluntary. Google generally follows its own robots.txt rules and tokens rather than third party extensions, so the signals work best alongside crawler level rules that actually allow or block specific bots.
How to choose your settings
The right combination depends on what you want from AI. A common setup for brands that want to be found and cited, while keeping content out of training, looks like this:
User-Agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: /
- Brands focused on being recommended: Keeping search and ai-input set to yes protects your AI visibility, since AI answers can only cite pages they are allowed to use.
- Publishers protecting their work: Setting ai-train=no, and possibly ai-input=no, limits reuse of content that is the product itself.
- Documentation and help content: Allowing all three often helps AI give accurate answers about your product.
After changing anything, the free AI crawler accessibility checker shows which AI bots can still reach your pages.
Curious whether AI is citing your pages?
MentionBird tracks your buying prompts every day across the AI platforms in your plan, and shows where your brand appears. Starter covers ChatGPT and Gemini, Growth adds Perplexity and Claude, and Custom supports up to 20 platforms: ChatGPT, Gemini, Perplexity, Claude, Grok, Google AI Overviews, Microsoft Copilot, DeepSeek, Google AI Mode, Mistral Vibe, Qwen Chat, Meta AI, Kimi AI, You.com, Ask Brave, Alexa for Shopping, Doubao, ERNIE, Poe, and Duck.ai.
Start todayOther glossary terms to check out
FAQs
1. Do I need Cloudflare to use Content Signals?
No. They are plain text lines in robots.txt, so any site can add them, whether or not it uses Cloudflare.
2. Will ai-input=no keep my site out of Google's AI Overviews?
Not reliably. Google has not committed to following Content Signals, so crawler level controls are still needed if this matters to you.
3. Can different sections of a site have different signals?
Some setups support rules for specific paths, such as allowing AI answers for documentation while restricting it elsewhere.
4. Does ai-train=no affect search rankings?
No. The signal only states a preference about training, and it does not change how search engines index or rank pages.
5. Are Content Signals the same as a license?
No. They state preferences in a machine readable way, while a license is a formal agreement about how content can be used.
See which of your pages AI cites
First month $19 with code TRY19, then $79/mo. Cancel anytime.
.png)
