Which AI crawlers should you allow in robots.txt?

01Why this one robots.txt decision matters more than most

Most robots.txt edits are low-stakes—a stray Disallow on a staging path, a sitemap line that’s slightly wrong. AI crawlers are different, because the wrong rule doesn’t just cost you a little crawl budget. It can quietly remove your business from the answer a customer is looking at right now, in ChatGPT, Perplexity, or Google AI Overviews.

The instinct a lot of site owners have—block anything with “bot” in the name that isn’t Google—comes from a real concern: nobody wants their content scraped to train someone else’s model for free. That concern is legitimate. It also leads people to block the wrong crawlers, because not every AI crawler does the same job, and treating them as one undifferentiated threat is how a business disappears from AI search by accident.

02The one distinction that actually matters: training vs. answering

Every major AI company that operates a crawler splits it into (at least) two roles:

The practical upshot: if your worry is “I don’t want my content used to train someone’s model,” you can act on that without giving up AI search visibility. Those are two different crawlers, not two names for the same thing.

03The crawlers, grouped by what they actually do

This is the same list we use on our own robots.txt (see it below), grouped by operator:

Not exhaustive—smaller AI products (Cohere’s cohere-ai, DuckDuckGo’s DuckAssistBot, and others) run their own crawlers too. The pattern holds regardless: check whether a given crawler trains or retrieves before deciding to block it.

04Should I block AI crawlers?

For most businesses trying to be found in AI search, the honest answer is no, allow them—and if you’re running our free scorecard or a similar audit, this is exactly the pillar it’s scoring. A retailer, a local service business, a B2B company that wants to show up when someone asks ChatGPT or Gemini for a recommendation needs the retrieval crawlers reading their site. Blocking them doesn’t protect anything; it just removes you from consideration.

There are narrower, legitimate reasons to block specific crawlers:

Outside those cases, blocking AI crawlers is usually a decision made out of a general unease about AI, applied with a tool that can’t distinguish “train on this” from “cite this when someone asks.”

05What does GPTBot control in robots.txt?

GPTBot is the crawler people ask about most, usually phrased as some version of “should I block GPTBot in robots.txt.” The specific answer: GPTBot only affects OpenAI’s training-data collection. It is not the crawler behind a live ChatGPT search answer.

That job belongs to two other user-agents—OAI-SearchBot, which OpenAI uses to index content for ChatGPT’s search feature, and ChatGPT-User, which fetches a specific page in the moment a user (or the assistant, acting on their behalf) requests it. Disallow GPTBot on its own and you’ve opted out of training data collection while leaving both of those retrieval paths untouched.

The mistake worth avoiding is disallowing User-agent: GPTBot when what you actually meant was every OpenAI crawler, or worse, disallowing User-agent: * to be safe and catching all three at once along with every other AI crawler that doesn’t have its own explicit rule.

06How to write the rule so it does what you mean

Give each AI crawler its own User-agent block rather than relying on the wildcard. This matters for a reason beyond clarity: our own scorer only counts a crawler as explicitly allowed when its directive isn’t User-agent: *—a wildcard allow states nothing about intent, where a named rule does. Sites with more than five AI crawlers explicitly named score meaningfully higher on crawler access than sites relying on the wildcard alone.

To allow the training and retrieval crawlers you want, and nothing else, name each one:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://yoursite.example/sitemap.xml

To opt a training-only crawler out while keeping its retrieval sibling allowed—the middle-ground option from the checklist above—just omit it or add an explicit Disallow:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

That combination trains nothing while still allowing ChatGPT to cite you in search results—the exact split most businesses actually want once they understand the distinction.

07A real example

Because we audit crawler access for a living, our own robots.txt names every major AI crawler explicitly rather than leaning on the wildcard:

User-agent: GPTBot
Allow: /
Disallow: /api/
Disallow: /clients
...

User-agent: ClaudeBot
Allow: /
Disallow: /api/
Disallow: /clients
...

User-agent: PerplexityBot
Allow: /
Disallow: /api/
Disallow: /clients
...

Same allow policy as our general User-agent: * rule, just repeated per crawler so each one is an explicit, intentional allow rather than an inferred one.

08Does one Disallow rule block all AI crawlers?

Yes, if it targets User-agent: * and no AI crawler has its own more specific rule above it—that wildcard catches every crawler in the table above that isn’t named individually. This is how most sites end up blocking AI crawlers without ever deciding to:

• A wildcard rule aimed at something else. User-agent: * Disallow: / on a staging environment that later goes live unchanged blocks every crawler with no more specific rule—including every AI crawler on the list above.
• Copying a “block AI scrapers” snippet from a forum. These almost always block both roles at once, since the snippet doesn’t distinguish GPTBot from OAI-SearchBot or ClaudeBot from Claude-Web.
• A CDN or security plugin’s default “bot protection.” Some ship with AI crawlers blocked by default at the WAF or CDN level, which won’t even show up as a rule in robots.txt—worth checking directly if your visibility looks worse than your audit score suggests it should.

09robots.txt and llms.txt are doing different jobs here

robots.txt is the file with actual enforcement power—it’s what determines whether a crawler is allowed to fetch a page at all. llms.txt is a curated map for crawlers that are already allowed in, telling them what matters once they arrive. Get the robots.txt rules wrong and nothing else you do for AI visibility can compensate—a blocked crawler never reads your schema, your content, or your llms.txt file, because it never reads the page.

See our guide to llms.txt for the file that matters once crawler access is sorted, and our GEO guide for how crawler access fits the bigger picture of getting cited in AI answers.

10The decision, in short

Allow the retrieval crawlers—OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User—unless you have a specific reason not to be found in AI search at all. Those are the ones actually deciding whether your business gets named.

The training crawlers—GPTBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent, anthropic-ai—are a genuine judgment call. Blocking them costs you nothing in today’s AI search results. Allowing them is a bet that being part of the next generation of models is worth more than the content itself.

What loses on every axis is a blanket rule that blocks all of them without knowing which is which—the outcome most site owners who block “AI bots” actually wanted least.