This week I opened my own site's robots.txt. Not the one in the repository. The one imlola.xyz actually serves. They aren't the same file.
The one I wrote starts with a comment saying the site is open to search engines and AI crawlers alike, and then lists GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more than a dozen others, each with Allow: /. The point of this site is to be read and cited, and that file says so explicitly. The version being served has an extra block on top, wrapped between two comments that read BEGIN Cloudflare Managed content and END Cloudflare Managed Content. That block declares Content-Signal: search=yes,ai-train=no,use=reference for every user agent, and then sets Disallow: / for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, CloudflareBrowserRenderingCrawler, Google-Extended, GPTBot and meta-externalagent.
Read top to bottom, the same file tells GPTBot not to crawl anything and then tells it to crawl everything. Search Console shows no problem. Nothing looks broken. That's exactly the kind of configuration that stays in place for years.
I'm writing about it now because tomorrow, September 15, Cloudflare changes the defaults behind that block, and the new rules have a side effect that goes well beyond one contradictory text file.
What Cloudflare changes on September 15
In a post published on July 1, Cloudflare replaced the old yes-or-no question about blocking AI bots with three categories defined by behavior. Search is "any behavior that collects or indexes your content, so it can answer questions about it later." Agent is "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now." Training is "a crawler taking your content to train or fine-tune a model."
Starting September 15, 2026, every new domain onboarding to Cloudflare gets Training and Agent blocked by default on pages that display ads, while Search stays allowed. The reasoning Cloudflare gives is compact: "An ad is a signal that a website owner meant for a person to land there and see it." Existing customers who don't want the new defaults can opt out from their Security settings any time before the date.
The weight of the change comes from scale. Cloudflare describes its reach as "the more than 20% of web domains that sit behind Cloudflare." When a network that size changes a default, it stops being a setting. It becomes a norm.
Up to here it reads like a reasonable measure to protect publishers from being crawled for free. The complication is in one sentence further down the same post.
Why Googlebot ends up on the blocked side
"Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training." That's Cloudflare's own wording, not a critic's interpretation.
The mechanism is simple once it's laid out. Cloudflare classifies Googlebot as a crawler that does both Search and Training. When a site blocks Training, the most restrictive rule wins, and the same crawler that indexes the site for Google Search gets blocked along with it. The consequence isn't a lower ranking. It's not being crawled at all.
A version of this had already surfaced before the date. In August, Search Engine Journal reported on a user in r/TechSEO who wrote that when they set AI Training to Block, "both Googlebot and Bingbot start receiving HTTP 403 responses when trying to fetch my sitemap." It's a single report, and SEJ itself noted it wasn't clear whether it was a bug, a configuration error, or the new rules arriving early. But it describes exactly the mechanism Cloudflare announced.
Cloudflare's position is that the fix belongs to the companies running the crawlers, and in the same post it strongly encourages them to separate crawlers by purpose. As of this writing, I haven't found an announcement from Google, Apple or Microsoft saying they've done it.
Google already draws that line, just somewhere else
The irony is that Google does have a separation. It just doesn't match Cloudflare's categories. According to Google's crawler documentation, Google-Extended is a token publishers can use to manage whether content Google crawls may be used to train future Gemini models and for grounding. The same page is explicit about what it doesn't do: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
What Google-Extended doesn't control is AI Overviews or AI Mode. Google's documentation on AI features says that to be eligible as a supporting link there, "a page must be indexed and eligible to be shown in Google Search with a snippet." To limit what shows up in those features, the tools are the same ones that limit a normal result: nosnippet, data-nosnippet, max-snippet and noindex.
So in Google's model, generative search is part of Search. That's consistent with what Google wrote in May in its guide to optimizing for generative AI features: "optimizing for generative AI search is optimizing for the search experience, and thus still SEO." In Cloudflare's model, Googlebot also does Training, so blocking Training reaches it. Each model is coherent on its own terms. They just don't fit together, and the site owner sits in the gap between them.
| Control | What it acts on | Requested or enforced | Effect on Google Search |
|---|---|---|---|
robots.txt Disallow for a specific bot | The crawler reads the rule and decides whether to follow it | Requested. RFC 9309 says these rules "are not a form of access authorization" | None, unless you disallow Googlebot itself |
Content-Signal in robots.txt | A declaration of permitted uses: search, ai-input, ai-train | Requested. Google's parser ignores rules other than allow, disallow and user-agent | None directly |
| Google-Extended token | Gemini training and grounding | Requested, documented by Google | None, per Google's documentation |
nosnippet, max-snippet, noindex | How the page appears in Search, AI Overviews and AI Mode included | Documented by Google as the control for AI features | Also reduces or removes your presence in regular results |
| Blocking Training in Cloudflare | The request itself, at the network, before it reaches your server | Enforced | Can block Googlebot, since Cloudflare treats it as multi-purpose |
What happens when a file says yes and no at the same time
Back to my robots.txt. The formal answer is in RFC 9309, the standard that defines the protocol. Section 2.2.1 says that if more than one group matches the same user agent, their rules "MUST be combined into one group." Section 2.2.2 says that when an allow rule and a disallow rule are equivalent, "the 'allow' rule SHOULD be used." Google documents the same behavior for its own crawlers: they merge the groups, and "in case of conflicting rules, Google uses the least restrictive rule."
Applied to my case, Google reads Google-Extended as allowed, because my Allow: / and Cloudflare's Disallow: / have the same path and the permissive rule wins. For every other crawler, the result depends on how each one implemented a SHOULD, and from the outside there's no way to verify it. A parser that stops at the first group matching its name would read Cloudflare's block and never reach mine.
The same logic explains why the Content-Signal line changes nothing for Google specifically. Google's parser only processes allow, disallow and user-agent, and ignores the rest. The signal still means something, but as a declaration of terms, not as a crawling rule Googlebot applies.
A contradictory robots.txt doesn't fail. It works differently for each reader. That's worse.
Why this is a GEO decision, not just a security setting
The impulse to block training is rational, and it's worth being clear about why. In an analysis Cloudflare published in 2025, covering June 19 to 26 of that year, Anthropic's crawlers made nearly 71,000 HTML page requests for every HTML page referral they sent back. Cloudflare added its own caveat: traffic from Claude's native app doesn't include a Referer header, so part of what comes back is invisible to that measurement. Even so, a publisher looking at a ratio like that, the most extreme version of the problem I described in zero-click is the new channel, has good reasons to want to close the door.
The problem is that AI isn't one door. OpenAI documents three separate agents. GPTBot crawls content that may be used to train its foundation models. OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features." ChatGPT-User visits a page when someone asks ChatGPT something that needs it. Blocking GPTBot tells OpenAI not to train on your content, but it isn't the switch for appearing in ChatGPT search. That one is OAI-SearchBot.
Anthropic splits its agents the same way. ClaudeBot collects web content that could contribute to training. Claude-SearchBot crawls to improve the quality of search results. Claude-User fetches pages when a person asks Claude a question. According to Anthropic, blocking those last two reduces a site's visibility in the answers users get.
That's where the whole decision sits. Training, search indexing and real-time retrieval are three different uses, and each company separates them differently. Cloudflare's managed robots.txt, to its credit, disallows a list of agents it associates with training, and doesn't include OAI-SearchBot, Claude-SearchBot or the user-triggered agents. That's a sensible design. The network-level Training block reaches further, because Googlebot doesn't have a training-only twin that could be blocked instead.
A few weeks ago I wrote that SEO and GEO aren't different channels, that they respond to the same signals with different emphasis. This is the same argument seen from the infrastructure side. For years, protecting content from AI and showing up in Google were two separate switches. On a site behind Cloudflare that chooses to block Training, they're now wired to the same one.
What to look at before and after September 15
The first thing is the robots.txt your domain actually serves, not the one in your repository or CMS. If Cloudflare's managed block is there, it starts with that BEGIN Cloudflare Managed content comment. Cloudflare's documentation confirms that it prepends its managed file "before your existing robots.txt, combining both into a single response." The option that controls it lives in Security settings, among the bot traffic options, as the preference to block training in robots.txt.
The second is separating two things that sound alike and aren't. A managed robots.txt asks. A Training block in AI Crawl Control enforces. The first can't stop Googlebot, because Googlebot isn't on its list and robots.txt is voluntary anyway. The second can, because of how Cloudflare classifies Googlebot.
The third is deciding what you're actually trying to prevent, because each goal has its own control. Keeping content out of Gemini training is what Google-Extended is for, without touching Search. Keeping content out of AI Overviews means snippet controls, with the cost of also changing how regular results look. Staying out of model training in general while still being cited in ChatGPT or Claude answers is the difference between GPTBot and OAI-SearchBot, or between ClaudeBot and Claude-SearchBot. None of those goals requires blocking a multi-purpose crawler at the network level.
In my case the decision had already been made, and written down in a comment at the top of the file: this site exists to be cited. What was missing was a check that the file being served matched the file that was written.
The default decides for the people who don't look
Cloudflare's reasoning about ads holds up, and pushing crawler operators to separate their uses is probably the right kind of pressure. But defaults aren't neutral. A new site that joins Cloudflare after tomorrow, with ads on its pages, starts from a configuration it didn't choose. And anyone who, for perfectly good reasons, switches Training to Block can end up blocking Googlebot along with it, with a sitemap returning 403 as one of the first visible signs.
Search Console will show it eventually. The robots.txt already shows part of the story today. You just have to open the one that's actually being served.