Three buckets, not one

Which AI crawlers matter - and which ones is your site accidentally blocking?

Most robots.txt files treat "AI bot" as one category and one decision. It isn't. GPTBot, OAI-SearchBot, and ChatGPT-User are three different crawlers doing three different jobs - and a rule written to block one on purpose often blocks the other two by accident.

28 Labs · August 2026

AI crawlers split into three buckets, not one: training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) that collect content for future model training; search/index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot) that build the indexes live AI answers pull from; and user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) that load a page in real time because a person asked a question. Blocking bucket one is an editorial choice with a slow, long-term cost. Blocking bucket two or three usually means you've cut yourself out of AI answers today, without meaning to.

There's no such thing as "the AI bot"

Server logs and CDN bot-management panels routinely lump ten-plus different crawlers under one label - "AI bot," "AI crawler," "GPT bot" - and hand you exactly one switch: allow or block. That's the wrong granularity. OpenAI alone runs three separate bots with three separate jobs. Anthropic runs three of its own. Every one of them can be allowed or disallowed independently in robots.txt, by its own user-agent token.

Treat them as one category and you're making a decision about your training-data exposure and your live AI-search visibility with the same lever - when those are two different bets with two very different payoffs.

The three buckets, and what blocking each one actually costs you

Bucket 1 - Training crawlers
GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple). These collect content - or, for Google-Extended and Applebot-Extended, tag data already gathered by the main crawler - for use in future model training. Blocking them is a legitimate, low-risk editorial call: OpenAI's documentation is explicit that GPTBot exists to make training data available, not to power live answers; Google says plainly that disallowing Google-Extended has no effect on Search inclusion or ranking. The cost of blocking is slow: future model versions know less about your content when describing your category. It does nothing to today's ChatGPT, Gemini, or Claude answers.
Bucket 2 - Search/index crawlers
OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), and Bingbot (Microsoft - not AI-labeled, but the index behind Copilot's own AI answers and a signal other engines blend in). These build the retrieval index the engine searches live when someone asks a question. Block one of these and you're not opting out of training - you're opting out of being citable on that engine, today. For most commercial sites this is self-harm: you've removed yourself from the exact moment a buyer is asking who to use.
Bucket 3 - User-triggered fetchers
ChatGPT-User (OpenAI), Claude-User (Anthropic), Perplexity-User (Perplexity). These don't crawl on a schedule. They fire once, live, because a specific person asked a specific question and the engine needs your page right now to answer it. OpenAI's own documentation says ChatGPT-User "is not used for crawling the web in any automatic fashion." Block it and you break the single highest-intent moment there is - a real person, mid-question, about to get an answer that could have come from your page.

GPTBot vs OAI-SearchBot: why blocking one doesn't touch the other

This is the confusion behind "should I block GPTBot" - people assume it also controls whether they show up in ChatGPT search citations. It doesn't. GPTBot and OAI-SearchBot are declared separately in OpenAI's documentation, respond to separate robots.txt tokens, and do separate jobs: GPTBot feeds training runs, OAI-SearchBot feeds ChatGPT's live search results. Disallow GPTBot and keep allowing OAI-SearchBot, and you keep showing up in ChatGPT's answers while opting out of training.

The same split holds at Anthropic (ClaudeBot vs Claude-SearchBot) and at Google, where Google-Extended governs Gemini and Vertex AI training exposure and has no bearing on whether Google or Bing indexes you for search - including the AI answers built on that index, which is worth reading alongside does ChatGPT use Bing or Google before you decide what to block where.

The 2026 AI crawler list, by bucket

One honesty note: Perplexity has documented that Perplexity-User can fetch a page a user directly pasted or named even when robots.txt disallows it, because the person explicitly asked for that exact URL. Worth knowing before you assume a disallow line is airtight everywhere.

The mistake sites keep making

The actual failure we see isn't a considered decision - it's a robots.txt or CDN bot-management ruleset written once, in a hurry, off a "block all AI bots" template, that doesn't distinguish GPTBot from OAI-SearchBot, or ClaudeBot from Claude-User. The site owner meant to keep content out of model training. What they actually did was disallow OAI-SearchBot and PerplexityBot too, and quietly stop appearing in ChatGPT and Perplexity answers. No error, no alert, nothing in analytics - a blocked crawler doesn't fail loudly, it just stops visiting.

The decision is per bucket, not per vendorDon't ask "do we allow AI bots." Ask three separate questions: do we want our content in future model training (bucket 1 - philosophy) - do we want to be citable in live AI answers (bucket 2 - revenue) - do we want to answer a real person's live question through the engine (bucket 3 - also revenue). Most commercial sites should default to allow on buckets 2 and 3, and treat bucket 1 as the only real either/or.

How to check what your site is actually doing

Two checks, both doable today. First, read your live robots.txt (yoursite.com/robots.txt) and your CDN or bot-management panel - Cloudflare, Akamai, and similar ship default "AI bot" or "AI scraper" categories that block by bucket-blind default, not by the choice you'd actually make reading this. Second, check server logs for the real user-agent strings above. A robots.txt rule tells you what you intended; logs tell you who's actually visiting and whether they're respecting it. If you've never looked, that mismatch is common enough that we treat it as a standard line item in an AI site readiness pass.

What we do about it

We check crawler access as one line item inside site readiness, not the whole diagnosis. Fixing robots.txt matters, but it only gets you back in the room - it doesn't tell you whether the AI actually names you once it's there. We pair the technical check with answer-level measurement: real buyer questions, asked across ChatGPT, Claude, Gemini, and Perplexity, on a schedule, so you know whether being crawlable turned into being recommended.

28 Labs measures how AI engines answer real buyer questions - and what moves the counts. Every number we publish re-derives to specific questions and engines. try28labs.com