AI Crawler Management: Control GPTBot & ClaudeBot

AI crawler management is no longer optional in 2026. Every site owner needs an intentional answer to one question: which AI bots are allowed to fetch your content, and for what purpose?

Three years ago the only crawler decisions that mattered were Googlebot and Bingbot. As of mid-2026, a working AI crawler policy has to address at least seven distinct user agents from five companies — and each one behaves differently. Block the wrong bot and you disappear from ChatGPT search citations. Allow the wrong bot and your premium content trains a competitor's model.

This guide walks through the full AI crawler robots.txt landscape: every major bot, its purpose, the exact directives that work, and the trade-offs of blocking versus allowing. It pairs with our llms.txt guide (curation for AI) and our agentic AI optimization playbook (preparing for AI agents that act on your site).

I run Verlua, a web development studio that ships AI-aware crawler configuration with every new build. The patterns below are what we deploy on client sites — not theory, but the exact blocks that pass current verification tests in May 2026.

The Seven AI Crawlers That Matter in 2026

Before writing a single rule, you need a clear inventory of what each crawler does. Bot names sound similar, but their behavior — and the cost of getting the rule wrong — is not.

GPTBot (OpenAI Training)

GPTBot is the dedicated training crawler from OpenAI, launched in August 2023 and documented at platform.openai.com/docs/bots. It fetches publicly accessible content and feeds it into the pipeline that trains future GPT model weights. Blocking GPTBot does not affect ChatGPT search citations because OpenAI uses a separate user agent for that.

OAI-SearchBot (ChatGPT Search)

OAI-SearchBot is the user agent that powers ChatGPT search results and citations. When a ChatGPT user runs a web-enabled query, this crawler fetches and indexes pages in real time, then the citation appears in the chat response. If your priority is showing up inside ChatGPT answers, this is the bot you must allow.

ChatGPT-User (Live Browsing)

ChatGPT-User fires when a single ChatGPT user clicks through to your page or asks ChatGPT to browse a specific URL. It is closer to a human visit than a crawl — a one-off fetch driven by an explicit user request. Most operators allow it because blocking it breaks the user's ability to share or read your page inside ChatGPT.

ClaudeBot (Anthropic)

ClaudeBot is Anthropic's training crawler -- it collects web content that may be used to train future Claude models. Anthropic has since added two more specialized agents: Claude-User, which fetches a page on demand when someone asks Claude to read or browse a specific URL, and Claude-SearchBot, which indexes content that can surface as a citation in Claude's answers. That means Anthropic now supports the same block-training-allow-citation split as OpenAI and Google. Anthropic documents all three crawlers at support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler.

PerplexityBot (Perplexity Index)

PerplexityBot crawls and indexes content that powers Perplexity's answer engine. Perplexity also operates Perplexity-User for live, user-initiated fetches. Blocking PerplexityBot removes you from Perplexity citations almost entirely. Perplexity publishes IP ranges and documentation at docs.perplexity.ai/guides/bots.

Google-Extended (Google AI Training)

Google-Extended is Google's opt-out user agent for AI training. Adding Disallow: / for Google-Extended stops Gemini, Vertex AI, and other Google AI products from training on your content. It does not affect Googlebot or Google AI Overviews — those still index and surface your pages through standard Google Search. Google documents the bot at developers.google.com/search/docs/crawling-indexing/overview-google-crawlers.

Applebot-Extended (Apple Intelligence)

Applebot-Extended is Apple's training opt-out, separate from the regular Applebot that powers Siri search and Spotlight. It was announced in mid-2024 and documented at support.apple.com/en-us/119829. Apple Intelligence and the on-device foundation models do not train on content where Applebot-Extended is disallowed.

The AI Crawler Decision Framework

Whether to block or allow a given bot depends on three questions about your content and business model. Run each crawler through this framework before writing rules.

1. 1. Is the content a competitive moat or a marketing surface? Original research, copyrighted writing, paid courses, and proprietary datasets are moats — blocking training crawlers protects them. Service pages, blog posts, and product descriptions are marketing surfaces — being visible inside AI answers is more valuable than the marginal cost of training.
2. 2. Does the bot directly generate visibility? Search and on-demand bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) create direct citation surface. Blocking them removes you from AI answers. Training bots (GPTBot, ClaudeBot, Google-Extended) do not create direct citations — their value is long-term presence in model knowledge.
3. 3. Is there a separate training and search agent? OpenAI, Google, and Anthropic all now split training from search-and-citation traffic, so you can protect training data on any of the three without losing citation surface.

For local service businesses in markets like Fresno — where competitors are still largely unaware of intentional AI crawler configuration — getting this policy right early creates a durable citation advantage. Verlua's AI SEO services in Fresno include AI-aware robots.txt setup as part of a comprehensive GEO visibility package.

AI Crawler robots.txt Recipes (Copy and Paste)

Below are four ready-to-paste robots.txt blocks that cover the most common policies. Use the one that matches your site type, then add your existing Googlebot, Bingbot, and sitemap rules at the bottom of the file.

Recipe 1: Allow Everything (Maximum AI Visibility)

Best for marketing sites, agencies, local services, and anyone whose primary content goal is being cited inside AI answers. No explicit allow rules are needed — robots.txt defaults to allow if no Disallow exists.

# robots.txt — Allow all AI crawlers
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

# Your existing rules
User-agent: *
Allow: /

Sitemap: https://www.yoursite.com/sitemap.xml

Recipe 2: Block Training, Allow Search (Most Common 2026 Setup)

Best for SaaS, B2B content sites, and publishers who want AI citations but do not want their content training future models. This blocks GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended while leaving search and on-demand bots open.

# robots.txt — Block training, allow search
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: *
Allow: /

Sitemap: https://www.yoursite.com/sitemap.xml

Anthropic now splits training from citation surface the same way OpenAI and Google do: ClaudeBot handles training, Claude-SearchBot handles indexing for citations, and Claude-User handles on-demand page fetches when a person asks Claude to read a specific URL. Disallow ClaudeBot and allow the other two to get training-data protection without losing Claude citations.

Recipe 3: Block Everything (Maximum Protection)

Best for paywalled publishers, original-research sites, and creators whose business model depends on content scarcity. This blocks all AI crawlers known as of mid-2026.

# robots.txt — Block all AI crawlers
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Anthropic-AI
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.yoursite.com/sitemap.xml

Recipe 4: Selective Block (Per-Directory)

Use this when you want AI crawlers to read marketing pages but never your premium content, customer dashboard, or research archive. The pattern mixes per-bot and per-directory rules.

# robots.txt — Selective per-directory block
User-agent: GPTBot
Disallow: /premium/
Disallow: /research/
Disallow: /members/
Allow: /

User-agent: ClaudeBot
Disallow: /premium/
Disallow: /research/
Disallow: /members/
Allow: /

User-agent: Google-Extended
Disallow: /premium/
Disallow: /research/
Disallow: /members/
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://www.yoursite.com/sitemap.xml

Implementing AI Crawler Rules in Next.js, WordPress, and Webflow

The robots.txt content is the same regardless of platform — the difference is how you generate and serve it. Below are platform-specific patterns.

Next.js (App Router)

Next.js 13+ supports a typed robots configuration at app/robots.ts that emits a robots.txt at build time. The advantage is type safety and version control. The downside is verbosity for many user agents — most teams switch to a raw file in public/robots.txt once the rule set grows past five blocks.

// app/robots.ts
import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: 'GPTBot', disallow: '/' },
      { userAgent: 'ClaudeBot', disallow: '/' },
      { userAgent: 'Google-Extended', disallow: '/' },
      { userAgent: 'Applebot-Extended', disallow: '/' },
      { userAgent: 'CCBot', disallow: '/' },
      { userAgent: 'OAI-SearchBot', allow: '/' },
      { userAgent: 'ChatGPT-User', allow: '/' },
      { userAgent: 'PerplexityBot', allow: '/' },
      { userAgent: 'Perplexity-User', allow: '/' },
      { userAgent: '*', allow: '/' },
    ],
    sitemap: 'https://www.yoursite.com/sitemap.xml',
  };
}

WordPress

Yoast, Rank Math, and AIOSEO all expose a robots.txt editor in their UI. For sites without an SEO plugin, edit the file directly via FTP or the cPanel file manager. WordPress generates a virtual robots.txt by default if no physical file exists — creating a physical file overrides the virtual one. Always test after publishing because cached responses can mask broken rules for hours.

Webflow, Framer, Squarespace, Shopify

Webflow exposes robots.txt under Site Settings > SEO. Framer offers an SEO panel in project settings. Squarespace and Shopify auto-generate robots.txt and historically did not allow full editing — Shopify added customization in 2023 via theme files (robots.txt.liquid). Confirm by fetching yoursite.com/robots.txt with curl and reading the actual response.

Server-Side Enforcement (For Hard Blocks)

robots.txt is voluntary. Well-behaved crawlers respect it. Bad actors ignore it. For sites with sensitive content, layer in server-side blocks at the edge or origin: check the User-Agent header, return 403 for matching bots, and rate-limit by IP for known crawler ranges. Cloudflare, Vercel, and Fastly all expose middleware for this. Here is a sketch in Vercel Edge Middleware:

// middleware.ts — Hard block selected AI bots
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';

const BLOCKED_AGENTS = [
  'GPTBot',
  'ClaudeBot',
  'CCBot',
  'Google-Extended',
];

export function middleware(req: NextRequest) {
  const ua = req.headers.get('user-agent') || '';
  if (BLOCKED_AGENTS.some(bot => ua.includes(bot))) {
    return new NextResponse('Forbidden', { status: 403 });
  }
  return NextResponse.next();
}

export const config = {
  matcher: '/((?!_next|favicon).*)',
};

Pair the middleware block with the equivalent robots.txt Disallow so well-behaved bots stop on the polite signal and bad actors hit the 403. For tactical depth on the broader technical layer, see our technical SEO audit playbook and website security essentials.

How to Verify Your AI Crawler Rules Are Working

Pushing the file is half the job. Verification confirms the rules parse, the right bots are blocked, and the right bots are still allowed.

1. 1. Fetch the live file with curl. Run curl https://yoursite.com/robots.txt and confirm the response is plain text matching what you intended. A 404 or HTML response means the file is not being served correctly.
2. 2. Validate syntax with a parser. Google's robots.txt tester (in Search Console) parses your file the same way Googlebot does. For AI crawlers, manually inspect each User-agent block for typos — "GPT-Bot" with a hyphen is not the same as "GPTBot".
3. 3. Check server logs for the bots you want to block. A week after publishing, grep your logs for the user agent strings. If GPTBot is still hitting pages where you disallowed it, the bot is either ignoring the rule (rare for OpenAI) or your rule has a syntax error.
4. 4. Check server logs for the bots you want to allow. If OAI-SearchBot is not appearing in logs at all, your sitemap or internal linking may be the bottleneck, not the robots.txt.
5. 5. Test AI search visibility. Run topical queries inside ChatGPT search and Perplexity. If your pages appear as citations, the search bots are working. If they do not appear, the issue is upstream of robots.txt — see the citation surface playbook.
6. 6. Document the policy publicly. Publish an AI usage policy page that mirrors the robots.txt intent in plain English. This signals editorial position to readers, partners, and platforms, and matters if a dispute over training data ever arises.

How the AI Crawler Mix Is Shifting in 2026

Rather than a single "percent of sites blocking" number, the clearest published data on this comes from crawl-purpose and traffic-share analysis, not a blocking-rate survey.

Two patterns stand out. First, dedicated search-purpose crawling nearly doubled year over year (8.82% to 16.45% of AI crawler traffic) while mixed-purpose crawling collapsed (53.23% to 36.96%) -- AI companies are increasingly running declared, purpose-specific crawlers instead of one bot for everything, which is exactly what makes a block-training-allow-search policy enforceable. Second, blocking is still inconsistent in practice: across the same network, only 7.9% of AI crawler requests drew an explicit 403 Forbidden in the first half of 2026, versus 70.4% that got a normal 200 OK. By individual bot, Googlebot (22.90%), Meta-ExternalAgent (14.04%), and ClaudeBot (11.88%) carried the largest share of AI-purpose traffic, ahead of GPTBot (8.34%) and Amazonbot (8.11%).

Six Common AI Crawler Management Mistakes

1. 1. Blocking GPTBot and assuming ChatGPT search is blocked too. They are separate agents. Blocking GPTBot has no effect on ChatGPT search citations. To remove yourself from ChatGPT search, block OAI-SearchBot.
2. 2. Blocking Googlebot when you meant to block Google-Extended. The two names look similar but do very different things. Googlebot powers standard Google Search and AI Overviews. Google-Extended is the AI training opt-out. Blocking Googlebot is a catastrophic SEO move; blocking Google-Extended is a routine policy choice.
3. 3. Typos in user agent strings. A robots.txt rule for "GPT-Bot", "gptbot", or "OpenAI-GPTBot" will not match OpenAI's actual crawler. The match is case-insensitive but spelling must be exact. Always copy the agent string from the company's official documentation.
4. 4. Forgetting to update robots.txt when a new bot launches. The bot list has changed every quarter since 2023. Set a recurring calendar reminder to review robots.txt every 90 days, or automate it with a build-time check that fails CI when a new known bot appears without a rule.
5. 5. Mixing allow and disallow without testing. Some parsers require the Allow rule before Disallow inside the same block. Others give precedence to the longest match. Test the actual rendered file with Google's tester and verify the parser matches your intent.
6. 6. Treating robots.txt as enforcement. robots.txt is a request, not a wall. Bad actors and emerging research crawlers may ignore it. For genuine protection of premium content, use authentication, paywalls, or server-side user-agent blocking — not just robots.txt.

A Worked Example: Applying the Framework

Here is how the framework plays out for a common profile: a law firm site with three priorities -- AI citation surface for its service pages, a proprietary case-results database it considers competitive content, and hesitation about whether allowing GPTBot would feed a competitor's legal AI tool.

A typical configuration for that profile: allow OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-SearchBot, Claude-User, and Googlebot site-wide. Disallow GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended site-wide. Add a per-directory Disallow for every AI crawler on /case-results/, so that database never enters the crawl even for the bots allowed everywhere else.

This is the shape most service-business sites land on: open the marketing surface to search and on-demand bots for citation reach, close training bots to protect the content that took real work to produce, and use a per-directory rule wherever there is a specific asset -- a proprietary database, an internal tool, a paid archive -- worth carving out on its own. Document the same intent in robots.txt and a public AI usage policy so it reads as editorial choice, not an oversight.

Where AI Crawler Management Is Heading

Three trends are reshaping the space in 2026 and are worth tracking as you design your policy.

• IETF AI Preferences (AI-PREFS). An IETF working group is drafting a standard for expressing AI training preferences via HTTP headers and well-known URIs, which would give site owners a more granular alternative to robots.txt. The draft is at datatracker.ietf.org/wg/aipref. Expect partial adoption in the next 12 to 18 months.
• Paid licensing markets. Cloudflare launched a Pay-Per-Crawl marketplace in July 2025 that lets site owners charge AI crawlers for access on a per-crawl basis. Reddit and the Associated Press have both signed direct data-licensing deals with OpenAI; other major publishers have taken the opposite path instead -- the New York Times sued OpenAI and Microsoft for copyright infringement in December 2023 rather than license its archive. High-value content is splitting between monetized access and litigation, not settling into one universal model.
• Agentic AI crawlers. Browse-on-behalf-of-user bots (ChatGPT-User, Claude-User) are growing faster than traditional indexers. The on-demand category will dominate by late 2026, which is why our agentic AI optimization guide treats it as a separate discipline.

Frequently Asked Questions