By Paul GosnellThe web crawling landscape has fundamentally changed. Where once you only needed to think about Googlebot and Bingbot, there are now over a dozen AI-specific crawlers from companies like OpenAI, Anthropic, Google, Apple, Meta, and Perplexity. Each crawler feeds a different AI product, and each has its own user-agent string, crawling behaviour, and respect for robots.txt directives.
Understanding these crawlers is essential for any website owner who wants to control how their content is used by AI systems. Whether you want to maximise your visibility across all AI platforms, selectively allow certain crawlers while blocking others, or opt out entirely, you need to know exactly which bots are visiting your site and what they do with your content.
This guide provides a complete, up-to-date reference for every major AI crawler, including their user-agent strings, the products they serve, their crawling behaviour, and how to configure access rules. Use the AgentReady scanner to audit which crawlers your site currently allows.
Understanding AI Crawler Categories
AI crawlers fall into three broad categories based on what they do with your content. Understanding these categories helps you make informed decisions about which bots to allow.
Training crawlers
Training crawlers collect web content to be included in large language model training datasets. Your content becomes part of the model's knowledge, allowing it to generate responses based on information from your site. The content is not stored verbatim but is distilled into model weights during training.
Key training crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google/Gemini), CCBot (Common Crawl), and Meta-ExternalAgent (Meta/Llama).
Search and retrieval crawlers
Search crawlers index your content for real-time retrieval. When a user asks a question, the AI system searches its index and retrieves relevant pages to generate an answer, often citing your URL as a source. This is the closest analogue to traditional search engine crawling.
Key search crawlers include OAI-SearchBot (OpenAI SearchGPT), PerplexityBot (Perplexity), and ChatGPT-User (real-time browsing in ChatGPT).
Browsing agents
Browsing agents fetch specific pages on demand when a user asks an AI to look something up. Unlike search crawlers that proactively index content, browsing agents only visit pages when triggered by a user request. They behave more like a user with a browser than a traditional crawler.
The primary browsing agent is ChatGPT-User, which OpenAI deploys when ChatGPT users enable the browsing feature.
OpenAI Crawlers
OpenAI operates three distinct crawlers, each serving a different purpose within the OpenAI ecosystem.
GPTBot
| Property | Value |
|---|---|
| User-Agent | GPTBot |
| Full UA String | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) |
| Purpose | Model training, search indexing |
| Products Fed | GPT models, ChatGPT, OpenAI API |
| Respects robots.txt | Yes |
| IP Ranges | Published at openai.com/gptbot-ranges.txt |
GPTBot is OpenAI's primary web crawler. It was first identified in August 2023 and has been actively crawling the web since. Content accessed by GPTBot may be used to train future GPT models and to index content for OpenAI's search products.
GPTBot is the most important AI crawler for most websites. If you only allow one AI crawler, make it GPTBot, because it feeds into the most widely used AI products globally.
# Allow GPTBot full access
User-agent: GPTBot
Allow: /
# Or block GPTBot entirely
User-agent: GPTBot
Disallow: /ChatGPT-User
| Property | Value |
|---|---|
| User-Agent | ChatGPT-User |
| Full UA String | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot) |
| Purpose | Real-time web browsing in ChatGPT |
| Products Fed | ChatGPT browsing feature |
| Respects robots.txt | Yes |
ChatGPT-User is deployed when a ChatGPT user asks the model to browse the web. Unlike GPTBot, it only visits specific pages that are relevant to the user's current conversation. Content accessed by ChatGPT-User is used to generate a response for that specific conversation and is not used for model training.
Allowing ChatGPT-User is generally low-risk because it only accesses content on demand and does not use it for training. Many sites that block GPTBot still allow ChatGPT-User for real-time referral traffic.
OAI-SearchBot
| Property | Value |
|---|---|
| User-Agent | OAI-SearchBot |
| Full UA String | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot) |
| Purpose | SearchGPT and ChatGPT search indexing |
| Products Fed | ChatGPT search, SearchGPT |
| Respects robots.txt | Yes |
OAI-SearchBot is the newest OpenAI crawler, launched alongside SearchGPT in late 2024. It indexes content specifically for OpenAI's search features, similar to how Googlebot indexes content for Google Search. Content accessed by OAI-SearchBot appears in ChatGPT search results with attribution and links back to your site.
This is the most directly valuable OpenAI crawler for traffic generation. SearchGPT results include links to source pages, driving direct referral traffic. If you allow any OpenAI crawler, prioritise OAI-SearchBot.
Anthropic Crawlers
ClaudeBot
| Property | Value |
|---|---|
| User-Agent | ClaudeBot |
| Full UA String | ClaudeBot/1.0 (https://www.anthropic.com/claubot; claubot@anthropic.com) |
| Purpose | Training data for Claude models |
| Products Fed | Claude (all versions), Anthropic API |
| Respects robots.txt | Yes |
ClaudeBot is Anthropic's web crawler, used to gather training data for Claude models. It has been active since early 2024. Anthropic publishes its crawling practices and provides an opt-out mechanism through robots.txt.
User-agent: ClaudeBot
Allow: /anthropic-ai
| Property | Value |
|---|---|
| User-Agent | anthropic-ai |
| Purpose | General Anthropic crawling |
| Products Fed | Anthropic research and products |
| Respects robots.txt | Yes |
The anthropic-ai user-agent is a secondary Anthropic crawler. It is less commonly seen than ClaudeBot but should be included in your robots.txt configuration for completeness. Allow both ClaudeBot and anthropic-ai to ensure full Anthropic compatibility.
Perplexity
PerplexityBot
| Property | Value |
|---|---|
| User-Agent | PerplexityBot |
| Full UA String | PerplexityBot/1.0 (+https://perplexity.ai/perplexitybot) |
| Purpose | Real-time search and answer generation |
| Products Fed | Perplexity.ai search engine |
| Respects robots.txt | Yes |
PerplexityBot powers Perplexity.ai, one of the most popular AI search engines. Unlike training crawlers, PerplexityBot indexes content for real-time retrieval. When a user asks Perplexity a question, it searches its index, retrieves relevant pages, and generates an answer with inline citations linking back to source pages.
Perplexity is one of the highest-value AI crawlers for traffic generation because it always provides source citations with clickable links. Allowing PerplexityBot directly translates to referral traffic.
Google AI Crawlers
Google-Extended
| Property | Value |
|---|---|
| User-Agent | Google-Extended |
| Purpose | Gemini AI model training |
| Products Fed | Gemini, Google AI features |
| Respects robots.txt | Yes |
| Affects Google Search | No |
Google-Extended is a crawler specifically for gathering training data for Google's Gemini AI models. It is completely separate from Googlebot. Blocking Google-Extended has zero impact on your Google Search rankings or indexing. This is one of the most commonly misunderstood crawlers.
If you want your content in Google Search but not in Gemini training data, block Google-Extended while allowing Googlebot. This is a perfectly valid configuration.
# Allow Google Search, block Gemini training
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /Googlebot (for AI Overviews)
Google AI Overviews (the AI-generated summaries at the top of search results) are powered by Googlebot, not Google-Extended. There is currently no way to appear in Google Search results but opt out of AI Overviews specifically. If you allow Googlebot, your content may appear in AI Overviews.
Apple Crawlers
Applebot-Extended
| Property | Value |
|---|---|
| User-Agent | Applebot-Extended |
| Purpose | Apple Intelligence and Siri features |
| Products Fed | Apple Intelligence, Siri, Safari suggestions |
| Respects robots.txt | Yes |
Applebot-Extended is Apple's AI-specific crawler, introduced alongside Apple Intelligence in 2024. It is separate from the standard Applebot (used for Siri and Safari suggestions). Allowing Applebot-Extended means your content may be used to train Apple's on-device AI models and power Apple Intelligence features across iPhone, iPad, and Mac.
Meta Crawlers
Meta-ExternalAgent
| Property | Value |
|---|---|
| User-Agent | Meta-ExternalAgent |
| Full UA String | Meta-ExternalAgent/1.0 (+https://www.facebook.com/externalhit_uatext.php) |
| Purpose | Meta AI training and features |
| Products Fed | Meta AI, Llama models, Facebook/Instagram AI features |
| Respects robots.txt | Yes |
Meta-ExternalAgent gathers training data for Meta's Llama models and Meta AI products. This includes AI features across Facebook, Instagram, WhatsApp, and Meta's standalone Meta AI product. Allowing this crawler means your content may appear in Meta AI responses across all Meta platforms.
Other Major AI Crawlers
Amazonbot
| Property | Value |
|---|---|
| User-Agent | Amazonbot |
| Purpose | Alexa AI features, Amazon search |
| Products Fed | Alexa, Amazon AI services |
| Respects robots.txt | Yes |
CCBot (Common Crawl)
| Property | Value |
|---|---|
| User-Agent | CCBot |
| Full UA String | CCBot/2.0 (https://commoncrawl.org/faq/) |
| Purpose | Open web dataset used by many AI labs |
| Products Fed | Common Crawl dataset (used by many AI companies) |
| Respects robots.txt | Yes |
CCBot is operated by Common Crawl, a non-profit that maintains a massive open dataset of the web. This dataset is widely used by AI labs for training, including by companies that may not operate their own crawler. Blocking CCBot reduces your content's availability across multiple AI training pipelines simultaneously.
Bytespider
| Property | Value |
|---|---|
| User-Agent | Bytespider |
| Purpose | ByteDance/TikTok AI features |
| Products Fed | TikTok AI, Doubao (Chinese AI assistant) |
| Respects robots.txt | Yes |
cohere-ai
| Property | Value |
|---|---|
| User-Agent | cohere-ai |
| Purpose | Cohere model training |
| Products Fed | Cohere Command, Cohere API |
| Respects robots.txt | Yes |
Diffbot
| Property | Value |
|---|---|
| User-Agent | Diffbot |
| Purpose | Structured data extraction, knowledge graph |
| Products Fed | Diffbot API, Knowledge Graph |
| Respects robots.txt | Yes |
YouBot
| Property | Value |
|---|---|
| User-Agent | YouBot |
| Purpose | You.com AI search |
| Products Fed | You.com search engine |
| Respects robots.txt | Yes |
Complete robots.txt Template for All AI Crawlers
Here is a comprehensive robots.txt configuration that explicitly allows all major AI crawlers. For a detailed syntax guide, see our robots.txt Guide.
# ===========================================
# robots.txt - Full AI Crawler Access
# Generated with p0stman AgentReady
# ===========================================
# Default rules
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /dashboard/
Disallow: /_next/
# OpenAI
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
# Anthropic
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
# Perplexity
User-agent: PerplexityBot
Allow: /
# Google AI
User-agent: Google-Extended
Allow: /
# Apple
User-agent: Applebot-Extended
Allow: /
# Meta
User-agent: Meta-ExternalAgent
Allow: /
# Amazon
User-agent: Amazonbot
Allow: /
# Common Crawl
User-agent: CCBot
Allow: /
# ByteDance
User-agent: Bytespider
Allow: /
# Cohere
User-agent: cohere-ai
Allow: /
# Diffbot
User-agent: Diffbot
Allow: /
# You.com
User-agent: YouBot
Allow: /
# Agent endpoints (allow for all bots)
User-agent: *
Allow: /api/mcp
Allow: /api/agent
Allow: /api/ai/context
Allow: /.well-known/agent.json
Sitemap: https://yourdomain.com/sitemap.xmlHow to Verify Which AI Crawlers Visit Your Site
Server access logs
The most reliable way to see which AI crawlers visit your site is to check your server access logs. Look for user-agent strings matching the known AI crawlers listed above.
# Search Nginx access log for AI crawlers
grep -E "GPTBot|ClaudeBot|PerplexityBot|ChatGPT-User|OAI-SearchBot|Google-Extended|Applebot-Extended|Meta-ExternalAgent|Amazonbot|CCBot|Bytespider|Diffbot|cohere-ai|YouBot" /var/log/nginx/access.logApplication-level bot tracking
For more detailed tracking, implement bot detection in your application middleware. Here is a Next.js middleware example that logs AI crawler visits to a database:
// middleware.ts
import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'
const AI_BOTS: Record<string, RegExp> = {
'GPTBot': /GPTBot/i,
'ChatGPT-User': /ChatGPT-User/i,
'OAI-SearchBot': /OAI-SearchBot/i,
'ClaudeBot': /ClaudeBot/i,
'PerplexityBot': /PerplexityBot/i,
'Google-Extended': /Google-Extended/i,
'Applebot-Extended': /Applebot-Extended/i,
'Meta-ExternalAgent': /Meta-ExternalAgent/i,
'Amazonbot': /Amazonbot/i,
'CCBot': /CCBot/i,
'Bytespider': /Bytespider/i,
'Diffbot': /Diffbot/i,
'cohere-ai': /cohere-ai/i,
'YouBot': /YouBot/i,
}
export function middleware(req: NextRequest) {
const ua = req.headers.get('user-agent') || ''
for (const [botName, pattern] of Object.entries(AI_BOTS)) {
if (pattern.test(ua)) {
// Log to your tracking endpoint
fetch(`${req.nextUrl.origin}/api/track-bot`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
bot_name: botName,
user_agent: ua,
path: req.nextUrl.pathname,
method: req.method,
country: req.headers.get('x-vercel-ip-country'),
}),
}).catch(() => {})
break
}
}
return NextResponse.next()
}AgentReady scanner
The p0stman AgentReady scanner provides an instant audit of your robots.txt configuration, showing which AI crawlers you currently allow and which you block. It also checks for related AI readiness signals like llms.txt, sitemaps, and MCP endpoints.
Impact on AI Search Visibility
ChatGPT and SearchGPT
Allowing GPTBot, ChatGPT-User, and OAI-SearchBot means your content can appear in ChatGPT conversations and SearchGPT results. ChatGPT has over 300 million weekly active users as of early 2026, making it one of the highest-traffic AI surfaces. SearchGPT results include source links, driving direct referral traffic.
Perplexity
Perplexity is one of the best AI sources for referral traffic because every answer includes inline citations with clickable links. Perplexity users tend to click through to sources at a much higher rate than traditional search users because the citations are integrated directly into the answer text.
Google AI Overview
Google AI Overviews appear at the top of Google Search results for many queries. They are generated from the same index as regular search results (via Googlebot), not from Google-Extended. You cannot opt out of AI Overviews without also opting out of Google Search entirely.
Claude
Claude's training data is gathered by ClaudeBot. While Claude does not currently offer a web search feature comparable to ChatGPT or Perplexity, allowing ClaudeBot means your content informs Claude's responses. As Claude adds search capabilities, being in the training data becomes increasingly valuable.
Frequently Asked Questions
How many AI crawlers are actively crawling the web in 2026?
At least 15 major AI crawlers are actively crawling the web as of March 2026, operated by OpenAI, Anthropic, Google, Apple, Meta, Amazon, Perplexity, ByteDance, Cohere, Diffbot, You.com, and Common Crawl. Each serves a different AI product or training pipeline. The number continues to grow as more companies launch AI products.
What is the difference between GPTBot, ChatGPT-User, and OAI-SearchBot?
GPTBot crawls content for model training and general indexing. ChatGPT-User browses web pages in real-time when a user enables browsing in ChatGPT. OAI-SearchBot indexes content specifically for SearchGPT and ChatGPT's search feature, similar to how Googlebot indexes for Google Search. Each can be independently allowed or blocked via robots.txt.
Does Google-Extended affect my Google Search ranking?
No. Google-Extended is completely separate from Googlebot. Blocking Google-Extended only prevents your content from being used for Gemini AI training. It has absolutely no impact on your Google Search rankings, indexing, or appearance in search results. You can safely block Google-Extended while maintaining full Google Search visibility.
Which AI crawlers should I allow for maximum visibility?
For maximum AI search visibility, allow all 15 major crawlers: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot, CCBot, cohere-ai, Bytespider, Diffbot, and YouBot. This ensures your content can appear across all AI-powered search surfaces.
How can I verify which AI crawlers are visiting my site?
Check your server access logs for user-agent strings matching known AI crawlers. You can also implement bot tracking in your application middleware to log crawler visits to a database, as shown in the Next.js middleware example in this guide. The AgentReady scanner provides a quick configuration audit.
Do AI crawlers respect robots.txt?
All major AI crawlers from reputable companies respect robots.txt directives. This includes crawlers from OpenAI, Anthropic, Google, Apple, Meta, Amazon, and Perplexity. They check your robots.txt file before crawling and honour Allow and Disallow rules for their specific user-agent strings.
What happens if I block ClaudeBot but allow GPTBot?
Your content will appear in ChatGPT and OpenAI search results but not in Claude's training data. Each AI crawler operates independently. Blocking one has no effect on the others. This granular control lets you create a fully customised access policy tailored to your business needs.
How do AI crawlers affect my server load?
AI crawlers typically crawl at moderate rates and respect Crawl-delay directives in robots.txt. Most sites see minimal impact on server performance. If you notice unusually high traffic from a specific bot, add a Crawl-delay directive for that user-agent. OpenAI and Anthropic publish rate information in their crawler documentation.