We checked 100 real sites: 3 in 10 block at least one AI crawler

We wanted a real number for a question we kept guessing at: how many sites actually block AI crawlers in robots.txt, and does it differ by industry? Not a vendor's marketing stat, not a LinkedIn post citing a LinkedIn post — an actual check of real robots.txt files, done with the same parser we ship in the free AI-crawler checker on this site.

The number

29.8% of the sites we checked block at least one of the 16 AI crawlers we track, at the site root. Only 1.1% block all of them. And 6.4% have no robots.txt file at all, which under the spec means they block nothing.

That headline number hides a much more interesting split by industry.

News sites are the outlier, by a wide margin

• News: 94.4% of the sites in our sample block at least one AI crawler.
• E-commerce: 16.7%.
• SaaS: 15.8%.
• Tech platforms: 15.0% (and this was the only category where any site — 5% of it — blocked every single crawler we track).
• Blogs and publishing platforms: 10.5%, the most open category in the sample.

Seventeen of eighteen news sites we checked block something. Zero of the wordpress.com, wordpress.org, dev.to, ghost.org, hashnode.com, or substack.com sites we checked block anything at all — the platforms writers actually publish on are, almost without exception, wide open.

That gap is the story. Publishers with subscription businesses to protect are drawing a hard line. Everyone else — including plenty of companies with just as much to lose from AI training on their content — mostly hasn't bothered.

Which bots get blocked

Across the 94 sites where we could read a robots.txt:

• ClaudeBot — 26.6% blocked, the most-blocked crawler in the sample.
• CCBot (Common Crawl) — 24.5%.
• Applebot-Extended — 22.3%.
• Meta-ExternalAgent — 20.2%.
• PerplexityBot and Amazonbot — 19.1% each.
• GPTBot — 18.1%.
• Google-Extended — 17.0%.
• bingbot — 1.1% and Applebot — 2.1%, barely touched.

The gap between bingbot (1.1%) and everything with "AI" or an LLM company's name attached to it (17–27%) is the clearest single result here: sites aren't blocking crawling in general, they're blocking crawlers they associate with AI training, specifically. bingbot still feeds Bing's regular search index — and, these days, a chunk of Copilot's answers — and almost nobody bothers to block it.

Worth flagging, since we've written about this mistake before: some of what we're counting as "blocked" here is a site blocking a search/retrieval crawler like OAI-SearchBot, Claude-SearchBot or PerplexityBot — which removes them from that assistant's live answers — not just a training crawler like GPTBot or ClaudeBot, which only opts out of model training and has no effect on citations. Our per-domain data (linked below) shows which is which for every site we checked; the headline percentages don't distinguish the two, and if you're deciding your own robots.txt, that distinction is the one that actually matters to your traffic.

Methodology, plainly

This is a hand-compiled convenience sample of 100 well-known domains split roughly evenly across five categories (news, e-commerce, blog/publishing platforms, SaaS, and tech platforms) — not a random sample of the web, not a scraped top-N ranking. It skews toward large, English-language, US/UK-headquartered organizations, and toward sites well-known enough to type from memory. Read the percentages above as directional for well-known, high-traffic sites — not as a claim about the median website, and not as a claim about sites in other languages or smaller markets.

Six domains were excluded from the percentages entirely rather than counted as either blocking or allowing: two returned a bot-protection challenge page instead of robots.txt (politico.com, searchengineland.com), and four failed to respond within our fetch window (npr.org, bestbuy.com, asos.com, webflow.com). A network failure isn't a data point about what a site's robots.txt says, so we left those out rather than guess.

"Blocking" means the site's robots.txt disallows that crawler at the root path (/). A site that blocks a bot from one folder but allows the rest doesn't count as blocking that bot here — it shows up in the raw data, but "blocks GPTBot on /paywall/" and "blocks GPTBot everywhere" are different claims and we didn't want to collapse them into one number.

We built the checker script to reuse the exact same parsing and matching logic — the same RFC 9309 group-precedence rules, the same 16-crawler list — as the free AI-crawler checker tool on this site, rather than writing a second, slightly different parser just for this research. If you run your own domain through that tool, you're getting the identical logic that produced these numbers.

Check your own site

The full per-domain results — which of the 16 crawlers each of the 94 sites blocks, and why the other six were excluded — are in the raw data behind this post if you want to dig into a specific industry or check whether a particular company blocks training but allows search (increasingly common) or the reverse (rarer, and usually a mistake — see the crawler roles piece for why that's backwards).