Robots.txt for AI Crawlers

Nearly every guide to AI crawlers is written as a guide to keeping them out. We publish more than 260 articles that GPTBot, ClaudeBot and PerplexityBot are all welcome to read, and the answers we get named in are a channel we never had to buy. Below, what each crawler actually does, the exact rules to copy, and the one line that deletes a company from AI answers without anyone noticing.

What Is Robots.txt for AI Crawlers?

The file itself has not changed. It is still a list of user agent names with Allow and Disallow rules under each one, served at yourdomain.com/robots.txt. What changed is who is reading it and why.

Until recently the only bots worth naming were search engine crawlers, and they all wanted the same thing. Now a single AI company runs three or four separate crawlers with three or four separate purposes, and the difference between them decides whether you show up when a buyer asks ChatGPT who the good agencies are. Most companies wrote their robots.txt years ago, have not opened it since, and have no idea which side of that line they are sitting on.

AI crawler

An automated bot run by an AI company that fetches public web pages. Some collect content to train models, some build a live index so an assistant can cite sources, and some fetch a single page on demand when a user asks the assistant to look at a URL. They are distinguishable only by the user agent string they send, which is the string you target in robots.txt.

Robots exclusion protocol

The convention that defines how robots.txt works, standardized as RFC 9309 in 2022. It defines the file format and the matching rules, and it describes directives that crawlers are requested to honor. It is not an access control mechanism, and nothing in the standard forces a crawler to obey it.

Two things follow from that definition, and both get missed. Robots.txt is permission, not protection, because a bot that ignores it faces no technical barrier. And robots.txt is permission, not visibility, because letting a crawler in does nothing on its own to make your pages worth quoting. We come back to both.

Which AI Crawlers Actually Matter?

There are four categories, and getting them straight is most of the work.

• Training crawlers collect content that may be used to train future models. GPTBot, ClaudeBot and CCBot are the main ones. Blocking these has no effect on whether you get cited today.
• Retrieval and search crawlers build the live index an assistant searches when it answers a question. OAI-SearchBot, Claude-SearchBot and PerplexityBot sit here. These are the ones that decide whether you can be cited at all.
• User-triggered agents fetch one page because a person asked the assistant to read it. ChatGPT-User, Claude-User and Perplexity-User. Blocking these means a buyer who pastes your URL into an assistant gets told the page cannot be read.
• Control tokens are not crawlers at all. Google-Extended is a name you can address in robots.txt that governs how already-crawled content may be used. Nothing fetches pages under that name.

Here is the full working list, with what each one costs you if you shut it out.

Every one of those names is documented by the company that runs it. OpenAI lists its crawlers and their full user agent strings in its bots documentation, Anthropic does the same in its crawler help article, Perplexity publishes both of its agents in its bot guide, and Google documents Google-Extended alongside every other token in its common crawlers reference. Read the primary source before you copy a rule off a blog. The list moves, and the third-party guides lag it by months.

Does Blocking GPTBot Remove You From ChatGPT?

No, and this is the single most expensive misunderstanding in the whole topic. GPTBot is the training crawler. ChatGPT search runs on OAI-SearchBot, a separate agent with a separate robots.txt token. A company that disallows GPTBot to keep its writing out of training runs is still fully eligible to be surfaced and linked when a buyer asks ChatGPT a question.

The reverse mistake is more common and much worse. Somebody reads a headline about AI companies scraping the web, adds a broad Disallow for every AI-sounding user agent they can find, and takes the company out of ChatGPT search, Claude search and Perplexity in one commit. Nothing breaks. No error appears anywhere. Traffic from those surfaces was never in the analytics to begin with, so nobody can see the hole that just opened.

Anthropic split its bots the same way, and the industry coverage of that change made the point plainly. Blocking ClaudeBot keeps you out of training. Blocking Claude-SearchBot keeps you out of Claude's search index. Those are different decisions with different consequences, and the file lets you make them separately.

Google-Extended is the third version of the same trap, running in the opposite direction. Plenty of teams disallowed it believing they were opting out of AI Overviews. They were not. Google-Extended governs training for Gemini. AI Overviews and AI Mode are Search features fed by Googlebot, so the only way out of them is to block Googlebot, which removes you from Google Search altogether. There is currently no way to appear in Search and not in AI Overviews. If that matters to your strategy, the details are in AI Overview optimization for B2B.

The rule worth memorizing: training bots decide whether you help build the model. Retrieval bots decide whether the model can name you. Only one of those shows up in your pipeline.

What Should Your Robots.txt Actually Say?

There are three defensible postures. Pick one deliberately instead of inheriting whatever the previous developer left behind.

The open posture is what most B2B companies should run, and it is what we run. Every AI crawler is named explicitly rather than left to the wildcard, because an explicit rule is unambiguous and a wildcard is a guess about how a given bot resolves precedence.

User-agent: *
Allow: /

# Training
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: CCBot
Allow: /

User-agent: Google-Extended
Allow: /

# Retrieval and search
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# User triggered
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

The selective posture is the one a publisher wants. It keeps your archive out of training sets while staying fully eligible for citations, which is the trade most media companies say they want when you ask them plainly.

# Keep the archive out of training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Stay eligible for citations
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Four syntax traps account for most of the broken files we look at.

1. The file is per host, and it does not inherit. A rule on yourdomain.com says nothing about blog.yourdomain.com or docs.yourdomain.com. Every subdomain needs its own file at its own root, which is exactly how a marketing team ends up with a wide open help center and a locked down blog without knowing it.
2. A named user agent block overrides the wildcard completely. If you write a block for GPTBot, GPTBot reads only that block and ignores everything under User-agent: *. A block containing nothing but a Crawl-delay line therefore wipes out the Allow you thought applied.
3. User agent names match on a token, not the full string. You write GPTBot, not the whole browser-style string the bot actually sends. Pasting the full user agent into the file matches nothing.
4. Blank lines separate groups. A stray blank line in the middle of a group splits it in two, and the rules below the break get attached to nothing. This is the failure that looks correct in a diff and is wrong in production.

Add the Sitemap line while you are in there. A crawler that respects robots.txt will read it, and it turns discovery of a 260 page corpus into one request instead of a link-following crawl that may never reach the deep pages. If you want that corpus retrieved as connected material rather than as isolated pages, pair it with the structure described in internal linking for LLM retrieval.

What Does Blocking Actually Cost You?

The case for blocking is real, and it is worth stating properly rather than waving away. Cloudflare put numbers on it by measuring how many pages each AI company crawls for every visitor it refers back. In their June 2025 analysis the ratios ran from roughly 70,900 crawl requests per referral at the extreme end down to companies sending back more traffic than they took, and Cloudflare noted that referrals from a native app do not carry a referer header, so the true ratios are likely friendlier than the raw figures. You can read the methodology in their crawl-to-refer analysis.

For a publisher, those ratios describe a genuine problem. The business monetizes the pageview, so an answer that quotes the article without sending the reader is pure extraction. Blocking training crawlers is a rational defense of an asset that took real money to produce.

For a B2B services company, the same ratio describes something closer to distribution. We do not monetize the pageview. We monetize the conversation that starts after a buyer decides we are worth talking to, and being named inside the answer where that decision happens is worth far more than the click we did not get. A 70,000 to 1 crawl ratio on a corpus that costs nothing extra to serve is not a cost, it is reach. That distinction is the whole argument, and it is unpacked further in LLM citation versus SEO traffic.

So ask one question before you touch the file. Does your revenue depend on the reader arriving, or on the reader deciding? If it is arriving, blocking training crawlers is defensible. If it is deciding, blocking anything is self-harm, and the reasoning behind which companies get named at all is in why AI answers cite some companies and not others.

Do AI Crawlers Actually Obey Robots.txt?

The major named ones do. The standard does not make them.

RFC 9309, published in 2022 and co-authored by the original creator of the protocol along with engineers at Google, is explicit that these rules are not a form of access authorization. It uses SHOULD rather than MUST throughout, and it describes directives that crawlers are requested to honor. Robots.txt is a sign on the lawn, not a lock on the door.

In practice the compliance picture is better than the standard implies. OpenAI, Anthropic, Perplexity and Google all publish that their crawlers honor the file, and they publish IP ranges so you can verify that a request claiming to be their bot really is. Anthropic serves a machine readable list at claude.com/crawling/bots.json for exactly that purpose. Common Crawl has honored robots.txt for CCBot since long before any of this was a story, and documents the behavior on its CCBot page.

What robots.txt cannot do is stop a bot that lies about who it is. Anyone can send a request with any user agent string. If you genuinely need to prevent access rather than request that it not happen, the file is the wrong tool and you want a firewall rule, rate limiting at the edge, or bot management from your CDN. Blocking in robots.txt and assuming the traffic stopped is the security equivalent of unplugging a monitor.

Two practical notes on verification. Do not trust user agent strings in your own logs without checking them against the published IP ranges, because a scraper impersonating GPTBot is common and will distort any read you take. And check what your file is doing today rather than what you believe it says, since the most common condition we find is a file written for a search-only web in 2021 that nobody has opened since.

Robots.txt Is Permission, Not Visibility

Here is the part that gets skipped. Allowing every crawler in does not get you cited. It gets you eligible. The gap between eligible and cited is where all the actual work lives, and no line in a text file closes it.

A model naming your company in an answer needs three things beyond access. It needs to be able to extract a claim cleanly, it needs to be confident the claim belongs to you specifically, and it needs a reason to prefer your version over the other 40 pages on the same question. Those map onto a short list of things you can build.

• Extractable blocks. A 50 word standalone summary near the top of a page gets lifted verbatim far more often than the same idea spread across four paragraphs. The pattern is in answer capsules explained, and the structural version is in comparison tables and AI citation.
• Machine readable structure. Schema tells a model what a page is rather than making it infer. Start with FAQ schema for AI visibility and, for a show, schema markup for podcast pages.
• A consistent entity. If your company name, description and category vary across your own pages, a model has to guess which version is true and often picks a competitor's cleaner description instead. The fix is in entity consistency for AI search and building an llm-info page.
• A stated summary of the site. The llms.txt proposal is a markdown file that hands a model a curated map of your site. Adoption is uneven, so treat it as a bonus rather than a substitute. What it is and how to write one is covered in llms.txt for B2B companies.
• Primary material worth quoting. Anyone can restate the same advice. Almost nobody publishes an original conversation with a named operator, which is why transcripts get cited so heavily. See podcast transcripts for AI search and turning episodes into answer content.

Do those and the whole practice has a name and a method, laid out in how to rank in AI search, GEO versus SEO, how to tune content for ChatGPT, and how to get cited by ChatGPT and Perplexity. The Perplexity specific version is here, and the list of formats models routinely skip is in what content AI assistants skip.

There is a useful parallel in the other text file that decides whether machines trust you. SPF, DKIM and DMARC are the same shape of problem in email: small, boring records at the root of a domain that quietly determine whether anything you send is believed. Nobody looks at them until delivery collapses. The mechanics are in what SPF, DKIM and DMARC are and email deliverability DNS records explained, and the reputation layer sitting on top is in what domain reputation is, what a secondary domain is, and how to set up email domains for outbound. Same lesson in both places. The config file nobody owns is the one that breaks the channel, and you find out months later from a number that was never there.

How to Audit Your Robots.txt in 10 Minutes

Run this now rather than scheduling it, because the failure mode here is silent and the check is short.

1. Read the live file. Open yourdomain.com/robots.txt in a browser, or run curl against it. Read the file that is being served, not the one in your repo, because a CDN or a platform can inject its own.
2. Repeat for every subdomain. Blog, docs, help, careers, app. Each one serves its own file and none of them inherit anything from the apex domain.
3. Search for Disallow. For each one, name the bot it applies to and say out loud what it costs you. If nobody can explain why a line is there, it was inherited rather than decided.
4. Check the three retrieval agents specifically. OAI-SearchBot, Claude-SearchBot and PerplexityBot. These are the ones that decide whether you can be cited today, and they are the newest, so they are the ones most likely to be missing from a file written a few years ago.
5. Check the user-triggered agents. ChatGPT-User, Claude-User and Perplexity-User. A buyer who pastes your URL into an assistant is the warmest reader you will get all week, and a stray Disallow means they get an error instead of your page.
6. Confirm the Sitemap line exists and resolves. Fetch the URL in it and confirm it returns the current sitemap rather than a 404 left over from a migration.
7. Pull the AI bots out of your server logs. Filter the last 30 days by user agent for GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot and PerplexityBot. Zero hits on a site that allows them usually means something upstream, a firewall rule or a CDN bot filter, is blocking what robots.txt permits.
8. Then measure the outcome, not the file. Ask the assistants your own category questions and record who gets named. The method is in how to audit your brand in ChatGPT and how to track AI search visibility. Expect 4 to 8 weeks between a fix and a change in the answers, since crawl, index and re-embed all lag, and plan your refresh cadence with how often to refresh content for AI search.

Ours is the open posture, published at highticketaisystems.com/robots.txt, with every crawler in the table above named explicitly and allowed. That is not generosity. Our buyers ask assistants who the good podcast agencies are, and a company the model cannot read is a company the model cannot recommend. The corpus those crawlers find is the same one described in what a podcast does for your search footprint and how to get your podcast cited by AI.

The Call We Would Make

If you sell a service to other businesses, allow everything, name every bot explicitly, add the Sitemap line, and go spend the saved afternoon on something a model would actually want to quote. The training question that dominates the discourse is close to irrelevant to you. Your content was never the asset. The conversations behind it were.

If you sell the content itself, run the selective posture. Disallow the training crawlers, allow the retrieval and user-triggered agents, and keep the citations. That is the trade most publishers say they want, and the file already supports it. Very few have configured it, because the split between the two kinds of bot is recent enough that most robots.txt files predate it.

What nobody should do is what most companies are currently doing, which is nothing. An unread file written for a search-only web is making a decision on your behalf every day, and it is a decision no one in the building would defend if you read it out in a meeting.

Open the file. It takes 10 minutes and it is the cheapest thing on any GEO list, precisely because it is a permission problem rather than a content problem. Then get back to the part that is actually hard, which is being worth citing once you have let everyone in.