A quarter of top sites are blocking AI crawlers: check robots.txt before rewriting content
Many brands are missing from AI answers not because their content is weak, but because their own robots.txt keeps AI crawlers out.
Key numbers
- Analysis of crawler restrictions across the top one million most-visited sites since 2023: about a quarter of the top 1,000 sites restrict AI crawlers, falling to roughly one in ten across the wider top million.
- Content type matters: 34.2% of news outlets disallow OpenAI's GPTBot, rising to 55% among outlets with high factual-reporting scores.
- Political leaning matters too: neutral outlets restrict the most (58%), while hyperpartisan and low-factual outlets restrict the least — only 4.1% of right-leaning outlets block OpenAI.
- The paper's inference: such uneven blocking can skew training data toward lower-quality or polarised content, affecting model behaviour downstream.
Method and sample
arXiv paper “Web Crawler Restrictions, AI Training Datasets & Political Biases” (2025-10-10): tracks robots.txt and crawler restrictions across the top one million most-visited sites since 2023, grouped by site popularity, content type and political leaning, with a discussion of downstream training-data effects.
What it means for brands
“AI doesn't cite us” is sometimes technical: one disallow line can stop AI from ever reading the page. And the crawlers differ — training crawlers versus search and answer crawlers — so blocking the first does not mean you should block the second.
How SourceSight handles it
The first step of the site audit is crawler access (can AI retrieve the site at all, is it public http/https), before schema and page-level signals; the generated llms.txt and site engineering files come with the same access guidance.
Limits
The study measures training crawlers such as GPTBot, which are not identical to search and answer crawlers, and skews to English-language sites, so it does not transfer directly to every market.
References
Run it on your own project
What the same prompts return across 10 engines is only knowable by running them.
Related research
Same engine, different surface, different answer: test app, web and API separately
On the same platform and the same questions, app and web cite systematically different source sets — and the API shows something else again.
Methods · 2026-10-07How to design a retest you can trust: five moves so noise does not pass as progress
AI visibility is a distribution, not a point: fix the prompts, repeat the runs, add controls, paraphrase, validate by hand — or you cannot attribute the change.