Should You Block AI Crawlers

27 May 2026

Blocking AI crawlers feels like the obvious move if you don’t want your writing used to train someone else’s model for free. But block the wrong ones and you also lose citation traffic from ChatGPT, Perplexity, and Google’s AI Overviews — the closest thing to organic search referral most sites still get. The two use cases are different bots. Treat them differently.

Training crawlers vs. citation crawlers

Training crawlers scrape your content to fine-tune or pretrain a model, with no promise of ever sending you a reader. Citation/retrieval crawlers fetch your page at answer-time, to ground a specific response and often link back to you as a source. Blocking both with one blanket rule throws away the traffic along with the training data.

Bot	Purpose	Blocking it costs you
`GPTBot`	OpenAI training	Training data only — no direct traffic link
`OAI-SearchBot`	Powers ChatGPT’s live web search/citations	Citation traffic from ChatGPT answers
`ClaudeBot` / `anthropic-ai`	Training and retrieval, varies by product	Mixed — check current docs, changes over time
`PerplexityBot`	Live retrieval for Perplexity answers	Citation traffic from Perplexity
`Google-Extended`	Opts you out of Gemini/AI Overviews training specifically, separate from regular Googlebot	AI Overviews inclusion, not regular search ranking
`CCBot` (Common Crawl)	Feeds a dataset used by many downstream model trainers	No direct traffic — pure training input

Bot names and their exact behavior change — verify current purpose against each company’s published crawler docs before you commit to a policy, don’t rely on a list from a year ago.

The decision for any given bot comes down to two questions:

flowchart TD
    A["Bot hits your site"] --> B{"Training or citation?"}
    B -->|Training only| C["Disallow"]
    B -->|"Citation (cites back)"| D{"Do you have a content moat to protect?"}
    D -->|Yes| C
    D -->|No| E["Allow"]

A defensible default policy

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Disallow: /

Block pure training crawlers. Allow the ones that fetch live and cite back — they’re closer to a search engine than a scraper, and they can send you readers.

The argument for blocking everything

If your business model depends on content being genuinely hard to reproduce elsewhere — a paid course, proprietary research, something you don’t want summarized for free — block all of it, training and citation bots alike. A citation in an AI answer that fully answers the question without a click is a citation that cost you the content and gave you nothing back.

The argument for allowing everything

If you’re a small blog and your actual goal is reach, being cited in an AI answer with a link is free distribution you didn’t have to earn through SEO. Most solo-dev and small-business blogs are in this camp whether they’ve thought about it or not — nobody’s scraping your Dokploy tutorial to build a competing product.

Check what’s actually hitting you before deciding

grep -iE "gptbot|claudebot|perplexitybot|ccbot|google-extended" access.log | awk '{print $1, $(NF-2)}' | sort | uniq -c

If you don’t have raw access logs, most reverse proxies (nginx, Caddy) can log user-agent on request — turn it on for a week before writing policy based on guesses.

The actual recommendation

For a developer or small-business blog with no proprietary moat: block pure-training bots (GPTBot, CCBot, Google-Extended), allow retrieval/citation bots (OAI-SearchBot, PerplexityBot). You keep the upside of being cited and cut off the crawlers that give you nothing back.

IAnswerable