The Short Answer
Block training crawlers if you want to — that is a legitimate business call. Keep search crawlers, because that is how you get cited. And think hard before blocking agents, because an agent request is usually a customer's request wearing different clothes. The mistake is not picking wrong on any one of these. It is applying one rule to all three.
Time-sensitive: September 15, 2026
From that date, Cloudflare's defaults change for new domains and for existing free customers who have not adjusted their settings: Search allowed, Training and Agent blocked on pages that display ads. Mixed-purpose crawlers that do not let you separate search from training from agent use will be blocked on ad-bearing pages entirely. If you run ads, check your dashboard before then so the policy is one you chose.
Why "Block AI" Stopped Being One Decision
Until recently, the practical question was binary: do you let AI companies crawl your content or not. Most site owners answered it once, in robots.txt, and moved on.
That framing broke because the traffic diversified. The same infrastructure now carries bots that index you so you can be cited later, bots that absorb your content into a model permanently, and bots that are acting for a specific human in real time. Those have entirely different value to you, and blanket rules treat them identically.
Cloudflare formalized the split on July 1, 2026, unifying what had been two separate concepts — verified bots and signed agents — and expressing the difference through metadata instead. Bots now carry a behavior classification (including Search, Agent, Training, and Transact) plus an operator label of Direct or Intermediary. You can read the live definitions in Cloudflare's verified bots documentation.
The Four Categories, and What Each Is Worth to You
| Category | What it does | Value to a business site | Default advice |
|---|---|---|---|
| Search | Indexes content to answer questions later, with a link | High — this is your citation path | Allow |
| Agent | Acts in real time on a specific person's behalf | High — often a live buyer | Allow |
| Training | Absorbs content into a model permanently | Low to negative for most | Your call |
| Transact | Completes a purchase or booking | High if you sell online | Allow |
Notice how lopsided that is for a normal business. Three of the four categories are things you want. Only training is genuinely arguable — and it is the one category where blocking has no downside to your own discovery.
Where publishers and businesses legitimately differ
If your business model is content — ad-supported publishing, subscriptions, a research library — the calculus changes. Your inventory is the thing being consumed, referral traffic is your revenue, and there is a live debate about compensation. That is why Cloudflare's new defaults are scoped specifically to pages that display ads: the platform is drawing the line where content monetization actually lives, not across the whole web.
A plumbing company, a law firm, or a Shopify store is in the opposite position. Your content is marketing collateral whose job is to get you found. Training crawlers taking it is a mild dilution; search and agent crawlers not reaching it is lost pipeline.
What Is Web Bot Auth, and Does It Change the Answer?
Web Bot Auth lets a bot prove cryptographically that it is what it claims to be. Rather than a user-agent string anyone can copy, each agent signs its requests using HTTP Message Signatures (RFC 9421), with a Signature-Agent header pointing to a published key directory. Cloudflare introduced the approach for signed agents in 2025, and it has been moving through the IETF as a draft standard with participation from major platform and AI vendors.
It is worth being precise about what this does and does not give you. Verification establishes identity. It does not establish permission, and it does not settle economics. A verified agent can still be blocked, rate-limited, or charged. What changes is that your policy can finally be specific — "allow this named agent, block that one" — instead of guessing from headers that anyone can forge.
A workable default policy for most business sites
- Allow search-purpose crawlers everywhere. You want the citations.
- Allow verified agents on public pages — product, pricing, service, contact, location.
- Block or challenge unverified automated traffic on login, checkout, and account endpoints.
- Decide deliberately on training crawlers, and write the decision into robots.txt as stated policy.
- Never put a CAPTCHA in front of a contact form, quote request, or product page.
- Rate-limit rather than block when volume is the concern — volume is a capacity problem, not an identity problem.
How Do You Audit What You Are Currently Blocking?
Most sites have never actually looked. Four checks, in order:
- Read your robots.txt out loud. Look for blanket
Disallow: /rules attached to AI user-agents that you added during the 2024–2025 blocking wave and never revisited. - Open your CDN or WAF bot settings. Find out whether you are on a managed default, and what that default currently does with the Agent category.
- Check for challenges on conversion paths. Any bot-protection rule sitting in front of a form, a pricing page, or a location page is costing you.
- Look at server logs, not just analytics. A JavaScript tag cannot record a visitor that got a 403 before the page loaded. Your logs can.
If you want the crawler-side mechanics in more depth — user-agent tokens, robots.txt syntax, and per-platform behavior — that is covered in our AI crawler management guide. This piece is the policy layer that sits on top of it.
The Argument for Blocking, Taken Seriously
It deserves a fair hearing rather than a strawman. The case runs like this: AI platforms consume enormous amounts of content and return very little traffic in exchange, and every answer delivered in-chat is a visit that never happens. If your content is the product, giving it away to a system that replaces you is not a growth strategy.
Cloudflare's own research popularized the crawl-to-refer ratio as a way to quantify this imbalance, and the direction of the numbers has been consistently unflattering to AI platforms. Publishers responding to that math by restricting training access are making a rational choice, and the emergence of pay-per-crawl and per-query compensation experiments — Cloudflare named partners including Ceramic.ai, You.com, beehiiv, Patreon, and Condé Nast in its 2026 announcement — suggests the market agrees the current exchange is lopsided.
None of that argues for blocking agents. An agent fetching three pages because your customer asked it to is not a crawl-to-refer problem. It is a session.
What To Do This Month
- Before September 15: open your CDN dashboard and confirm your bot policy is set intentionally, especially if you serve ads or onboarded recently.
- Separate your rules by purpose rather than by vendor. "Block training" is a policy; "block OpenAI" is a blunt instrument that also removes you from ChatGPT's search results.
- Remove bot friction from conversion paths. This is usually the single highest-value change.
- State your policy in robots.txt so it is legible to operators that honor it.
- Re-test with an AI browser afterwards to confirm a real agent can still complete a real task. Our guide to AI browser traffic has the four-task test we use.
Frequently Asked Questions
Should I block AI bots from my website?
It depends entirely on which kind, and "AI bots" is too coarse a category to answer. Blocking training crawlers is a defensible business decision with real arguments on both sides. Blocking search crawlers usually costs you citations and referral traffic you would rather have. Blocking agents means blocking software that a specific customer sent to your site to do something — which is almost never what a business wants. One rule for all three is how site owners accidentally cut off their own buyers.
What changes on September 15, 2026?
Cloudflare has announced new defaults from that date. New domains onboarding to the network — and existing free customers who have not adjusted their settings — will allow Search access while blocking Training and Agent categories on pages that display ads. Separately, crawlers that mix search, agent, and training purposes into a single bot without letting site owners distinguish them will be blocked on ad-bearing pages. Settings remain changeable in the dashboard at any time, but the default becomes an active choice you should make rather than inherit.
What is the difference between Direct and Intermediary bots?
It is Cloudflare's operator label, added when it unified verified bots and signed agents on July 1, 2026. A Direct bot is operated by a single narrow operator, usually on that operator's own infrastructure — a classic search crawler, for example. An Intermediary is an agentic service that a wide range of end users can drive, meaning any given request represents some person's intent rather than the vendor's. The distinction matters because it tells you whether a block hits a company or a customer.
What is Web Bot Auth?
Web Bot Auth is a mechanism for a bot to cryptographically prove its identity instead of relying on a spoofable user-agent string or a brittle IP allowlist. It uses HTTP Message Signatures (RFC 9421) with a per-agent key and a Signature-Agent header pointing at a published key directory. Cloudflare introduced it in 2025 and it has been progressing through the IETF as a draft standard. Verification is not authorization: proving who is calling does not decide whether they are allowed in, or at what price.
Does robots.txt still work for AI crawlers?
For well-behaved operators, yes — the major AI companies publish user-agent tokens and largely honor directives. But robots.txt is a request, not an enforcement mechanism, and it cannot distinguish between purposes when a single crawler serves several. That is exactly the gap the new taxonomy and signature-based verification are trying to close. Use robots.txt as your stated policy and network-level rules as your enforcement.
Will blocking AI crawlers hurt my SEO?
Blocking a search-purpose crawler will reduce your chance of being cited in AI answers, which is increasingly where discovery happens. Blocking training crawlers has no direct effect on classic Google rankings, since Googlebot indexing and AI training are separable for most policy decisions. The risk to watch is over-broad rules — a blanket block that catches search-purpose or agent traffic alongside training crawlers costs visibility you did not mean to give up.
Sources
- Cloudflare — Verified bots documentation (Direct vs Intermediary, behavior categories)
- Cloudflare — "Your content, your rules" press release, including September 15, 2026 defaults
- Cloudflare — The age of agents: cryptographically recognizing agent traffic
- Cloudflare — The crawl before the fall of referrals (crawl-to-refer ratio methodology)
Not sure what your site is currently blocking?
We audit bot policy alongside technical SEO — finding the rules that are quietly costing you citations, agent traffic, and form fills, then setting a policy you actually chose.
Related Articles
Founder & Technical Director
Mark Shvaya runs Verlua, a web design and development studio in Sacramento. He builds conversion-focused websites for service businesses, e-commerce brands, and SaaS companies.
California real estate broker, property manager, and founder of Verlua.