How to Track AI Bot and Crawler Activity

Most website owners know how much human traffic they receive. They have Google Analytics, Vercel Analytics, or Plausible tracking every page view. But they have no idea how much bot traffic they receive, which AI crawlers are visiting, or what those crawlers are looking at.

This is a blind spot. In 2026, AI crawlers from OpenAI, Anthropic, Google, Perplexity, Apple, Meta, and others are visiting websites millions of times per day across the web. Whether they visit your site, and what they find when they do, directly determines whether your content appears in ChatGPT responses, Claude conversations, Perplexity search results, Google AI Overviews, and Apple Intelligence features.

If you are not tracking this traffic, you are flying blind. You have no way to know if your robots.txt configuration is working, whether your llms.txt file is being discovered, or if your content is being indexed at all.

This guide covers everything you need to implement comprehensive bot tracking: all the major AI bot user-agent strings, Next.js middleware detection, Supabase database logging, an API route for recording visits, and SQL queries for building a bot analytics dashboard. Use the AgentReady scanner to check which bots can access your site.

Why tracking bot visits matters

Bot tracking serves three critical purposes for AI visibility:

1. Verify your discovery layer is working

You have configured your robots.txt to allow AI crawlers, created an llms.txt file, added structured data, and submitted your sitemap. But are the bots actually coming? Without tracking, you have no way to confirm. Bot tracking data shows you definitively whether GPTBot, ClaudeBot, and PerplexityBot are visiting your site and how often.

2. Understand what content AI systems value

Different bots crawl different pages with different frequency. By tracking which pages each bot visits most, you learn what content the AI systems consider most valuable. This tells you where to invest in deeper content, better structured data, and stronger answer capsules.

3. Detect problems early

If GPTBot was visiting daily and suddenly stops, something changed. Maybe a robots.txt update accidentally blocked it. Maybe a server error is preventing crawling. Without historical tracking data, you would never notice until your AI search traffic dropped weeks later.

Complete AI bot user-agent reference

Here is a comprehensive reference of every major AI and search bot user-agent string you should track. This list covers all the bots that matter for AI search visibility as of March 2026.

Bot Name	User-Agent String	Company	Purpose
GPTBot	GPTBot	OpenAI	Training data and search indexing
ChatGPT-User	ChatGPT-User	OpenAI	Real-time web browsing in ChatGPT
OAI-SearchBot	OAI-SearchBot	OpenAI	ChatGPT Search indexing
ClaudeBot	ClaudeBot	Anthropic	Content indexing for Claude
anthropic-ai	anthropic-ai	Anthropic	AI research crawling
PerplexityBot	PerplexityBot	Perplexity	Real-time search answering
Google-Extended	Google-Extended	Google	Gemini AI training data
Googlebot	Googlebot	Google	Search indexing and AI Overviews
Applebot-Extended	Applebot-Extended	Apple	Apple Intelligence features
Applebot	Applebot	Apple	Siri and Spotlight search
Bytespider	Bytespider	ByteDance	TikTok search and AI features
Meta-ExternalAgent	Meta-ExternalAgent	Meta	AI training for Meta products
FacebookBot	FacebookBot	Meta	Link previews and indexing
Amazonbot	Amazonbot	Amazon	Alexa and Amazon search
cohere-ai	cohere-ai	Cohere	AI model training
CCBot	CCBot	Common Crawl	Open web corpus (used by many AI labs)
Diffbot	Diffbot	Diffbot	Structured data extraction
YouBot	YouBot	You.com	AI search indexing
Bingbot	bingbot	Microsoft	Bing search and Copilot

Database schema for bot tracking

Create a bot_crawls table in Supabase to store bot visit data. This schema captures everything you need for analytics:

-- Supabase migration: bot_crawls table
CREATE TABLE bot_crawls (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  bot_name text NOT NULL,
  user_agent text,
  path text NOT NULL,
  method text DEFAULT 'GET',
  country text,
  created_at timestamptz DEFAULT now()
);

-- Index for time-based queries (most recent crawls)
CREATE INDEX idx_bot_crawls_created_at
  ON bot_crawls (created_at DESC);

-- Index for bot-specific queries
CREATE INDEX idx_bot_crawls_bot_name
  ON bot_crawls (bot_name);

-- Index for path-based queries
CREATE INDEX idx_bot_crawls_path
  ON bot_crawls (path);

-- RLS: allow inserts from the API (service role)
ALTER TABLE bot_crawls ENABLE ROW LEVEL SECURITY;

CREATE POLICY "Allow service role full access"
  ON bot_crawls FOR ALL
  USING (auth.role() = 'service_role');

CREATE POLICY "Allow anon inserts"
  ON bot_crawls FOR INSERT
  WITH CHECK (true);

Bot detection in Next.js middleware

The middleware runs on every request and is the ideal place to detect bot traffic. It runs on Vercel's Edge runtime, which means it executes before your page renders and adds minimal latency.

Bot detection map

Define a map of user-agent substrings to bot names. The middleware checks each incoming request's user-agent against this map:

// src/lib/bot-detection.ts
export const BOT_USER_AGENTS: Record<string, string> = {
  GPTBot: "GPTBot",
  "ChatGPT-User": "ChatGPT-User",
  "OAI-SearchBot": "OAI-SearchBot",
  ClaudeBot: "ClaudeBot",
  "anthropic-ai": "anthropic-ai",
  PerplexityBot: "PerplexityBot",
  "Google-Extended": "Google-Extended",
  "Applebot-Extended": "Applebot-Extended",
  Applebot: "Applebot",
  Bytespider: "Bytespider",
  "Meta-ExternalAgent": "Meta-ExternalAgent",
  FacebookBot: "FacebookBot",
  Amazonbot: "Amazonbot",
  "cohere-ai": "cohere-ai",
  CCBot: "CCBot",
  Diffbot: "Diffbot",
  YouBot: "YouBot",
  bingbot: "Bingbot",
  Googlebot: "Googlebot",
};

export function detectBot(
  userAgent: string
): { name: string; fullAgent: string } | null {
  for (const [pattern, name] of Object.entries(BOT_USER_AGENTS)) {
    if (userAgent.includes(pattern)) {
      return { name, fullAgent: userAgent };
    }
  }
  return null;
}

Middleware implementation

// src/middleware.ts
import { NextRequest, NextResponse } from "next/server";
import { detectBot } from "@/lib/bot-detection";

export function middleware(req: NextRequest) {
  const userAgent = req.headers.get("user-agent") || "";
  const bot = detectBot(userAgent);

  if (bot) {
    // Get geo data from Vercel headers (free on Vercel)
    const country = req.headers.get("x-vercel-ip-country") || "";

    // Fire-and-forget: log the bot visit without blocking the request
    const trackUrl = new URL("/api/track-bot", req.url);

    fetch(trackUrl.toString(), {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        bot_name: bot.name,
        user_agent: bot.fullAgent,
        path: req.nextUrl.pathname,
        method: req.method,
        country,
      }),
    }).catch(() => {
      // Silent fail - never block a bot request for logging
    });
  }

  return NextResponse.next();
}

export const config = {
  matcher: [
    // Match all paths except static files and internal Next.js paths
    "/((?!_next/static|_next/image|favicon.ico|.*\\.(?:svg|png|jpg|jpeg|gif|webp|ico|css|js|woff|woff2|ttf|eot)$).*)",
  ],
};

Important implementation notes

• Never block bot requests. The middleware must always call NextResponse.next() to allow the bot to access the page. Blocking would defeat the purpose of having AI crawlers visit your site.
• Use fire-and-forget logging. The fetch() call to the tracking API is not awaited. The middleware continues processing immediately. If the tracking call fails, it fails silently.
• Check order matters. The bot detection map checks more specific strings first. For example, Applebot-Extended is checked before Applebot so the more specific match wins.
• Vercel geo headers. The x-vercel-ip-country header provides the visitor's country code for free on Vercel deployments. This data is not available on localhost.

API route for logging bot visits

The middleware sends bot data to this API route, which inserts it into Supabase:

// app/api/track-bot/route.ts
import { NextRequest, NextResponse } from "next/server";
import { createClient } from "@supabase/supabase-js";

// Use service role for server-side inserts
const supabase = createClient(
  process.env.NEXT_PUBLIC_SUPABASE_URL!,
  process.env.SUPABASE_SERVICE_ROLE_KEY!
);

export async function POST(req: NextRequest) {
  try {
    const body = await req.json();

    const { bot_name, user_agent, path, method, country } = body;

    if (!bot_name || !path) {
      return NextResponse.json(
        { error: "bot_name and path required" },
        { status: 400 }
      );
    }

    await supabase.from("bot_crawls").insert({
      bot_name,
      user_agent: user_agent?.slice(0, 500) || null,
      path: path.slice(0, 2000),
      method: method || "GET",
      country: country || null,
    });

    return NextResponse.json({ ok: true });
  } catch {
    return NextResponse.json({ ok: false }, { status: 200 });
  }
}

Throttling high-frequency bots

Some bots like Googlebot and Bingbot make hundreds of requests per hour. To avoid overwhelming your database, you can add simple throttling:

// Simple in-memory throttle (resets on cold start)
const recentLogs = new Map<string, number>();
const THROTTLE_MS = 5000; // Log at most once per 5 seconds per bot+path

export async function POST(req: NextRequest) {
  const body = await req.json();
  const { bot_name, path } = body;

  // Throttle check
  const key = `${bot_name}:${path}`;
  const lastLog = recentLogs.get(key) || 0;
  const now = Date.now();

  if (now - lastLog < THROTTLE_MS) {
    return NextResponse.json({ ok: true, throttled: true });
  }

  recentLogs.set(key, now);

  // Proceed with database insert...
}

Building a bot analytics dashboard

Once you have bot tracking data accumulating, you can build queries to understand your AI crawling patterns. Here are the essential analytics queries.

Total crawls by bot

SELECT
  bot_name,
  COUNT(*) as total_crawls,
  MIN(created_at) as first_seen,
  MAX(created_at) as last_seen
FROM bot_crawls
GROUP BY bot_name
ORDER BY total_crawls DESC;

Daily crawl trends (last 14 days)

SELECT
  DATE(created_at) as date,
  bot_name,
  COUNT(*) as crawls
FROM bot_crawls
WHERE created_at > NOW() - INTERVAL '14 days'
GROUP BY DATE(created_at), bot_name
ORDER BY date DESC, crawls DESC;

Most crawled pages

SELECT
  path,
  COUNT(*) as total_crawls,
  COUNT(DISTINCT bot_name) as unique_bots,
  array_agg(DISTINCT bot_name) as bots
FROM bot_crawls
GROUP BY path
ORDER BY total_crawls DESC
LIMIT 20;

Pages that specific bots visit most

-- What is GPTBot most interested in?
SELECT
  path,
  COUNT(*) as crawls
FROM bot_crawls
WHERE bot_name = 'GPTBot'
GROUP BY path
ORDER BY crawls DESC
LIMIT 20;

Crawl frequency by hour of day

SELECT
  EXTRACT(HOUR FROM created_at) as hour,
  bot_name,
  COUNT(*) as crawls
FROM bot_crawls
WHERE created_at > NOW() - INTERVAL '7 days'
GROUP BY EXTRACT(HOUR FROM created_at), bot_name
ORDER BY hour, crawls DESC;

Geographic distribution of bot requests

SELECT
  country,
  COUNT(*) as crawls,
  array_agg(DISTINCT bot_name) as bots
FROM bot_crawls
WHERE country IS NOT NULL
GROUP BY country
ORDER BY crawls DESC
LIMIT 10;

Pages crawled by AI bots but not traditional search

-- Find pages that AI bots care about but Googlebot does not
SELECT path, COUNT(*) as ai_crawls
FROM bot_crawls
WHERE bot_name IN ('GPTBot', 'ClaudeBot', 'PerplexityBot')
  AND path NOT IN (
    SELECT DISTINCT path FROM bot_crawls WHERE bot_name = 'Googlebot'
  )
GROUP BY path
ORDER BY ai_crawls DESC
LIMIT 20;

Using bot data to optimise your discovery layer

The real value of bot tracking is not the data itself but the actions you take based on it.

If a bot is not visiting at all

Check your robots.txt. You may have accidentally blocked the bot. Verify your file allows the specific user-agent. Also check that your site is reachable (no 5xx errors, no firewall blocking the bot's IP range).

If bots visit your homepage but not content pages

Your internal linking may be weak. Ensure your homepage links to your key content pages. Submit a comprehensive sitemap and use IndexNow to proactively notify search engines about new content.

If bots visit pages without answer capsules

Prioritise adding <div class="answer-capsule"> elements to the pages bots visit most. AI models cite the first 30% of content 44% of the time, so putting a clear answer at the top of your most-crawled pages maximises your chances of being cited.

If crawl frequency drops suddenly

Something changed. Common causes: a robots.txt edit that accidentally blocked a bot, a server error (5xx) that made the bot back off, a DNS change that affected crawl routing, or rate limiting that is too aggressive. Check your server logs alongside the bot tracking data to diagnose.

If one bot visits far more than others

This is normal. Different bots have different crawl frequencies. Googlebot is typically the most frequent, followed by Bingbot. AI-specific bots like GPTBot and ClaudeBot tend to crawl less frequently but access more diverse pages. Focus on ensuring all the bots you care about are visiting at least weekly.

Live bot activity feed

For a real-time view of bot activity on your site, you can build a live feed component. This is useful for dashboards and for demonstrating AI crawling activity to stakeholders:

// Query for recent bot activity (last 24 hours)
SELECT
  bot_name,
  path,
  country,
  created_at
FROM bot_crawls
WHERE created_at > NOW() - INTERVAL '24 hours'
ORDER BY created_at DESC
LIMIT 50;

You can poll this query every 30 seconds from a dashboard component or use Supabase's real-time subscriptions for instant updates.

Privacy and compliance considerations

Bot tracking raises fewer privacy concerns than human visitor tracking because bots are not natural persons and are not covered by GDPR, CCPA, or similar privacy regulations. However, keep these points in mind:

• Do not store IP addresses. There is no need to store bot IP addresses for analytics. The user-agent string, path, and timestamp provide all the insight you need.
• Country-level geo data is sufficient. The Vercel x-vercel-ip-country header provides country codes, not precise locations. This is adequate for understanding geographic crawl patterns.
• Truncate user-agent strings. Some user-agent strings can be very long. Truncate to 500 characters to keep your database clean.
• Set a retention policy. Consider deleting bot tracking data older than 90 days to keep your database size manageable. Historical trends can be preserved as aggregated daily summaries.

Integration with other guides

Bot tracking is one component of a comprehensive AI visibility strategy. Here is how it connects to the other pieces:

• robots.txt controls which bots can access your site. Bot tracking verifies that your rules are working as intended.
• llms.txt provides AI-specific content discovery. Bot tracking shows whether bots are accessing your llms.txt file.
• IndexNow notifies search engines about new content. Bot tracking confirms that the notification led to a crawl visit.
• CORS headers enable cross-origin agent requests. Bot tracking helps you understand which agents are making those requests.
• Contact forms provide action endpoints for agents. Bot tracking shows which bots visit your contact page.

Frequently Asked Questions

Why should I track AI bot visits to my website?

Tracking AI bot visits tells you which AI systems are crawling your content, how often they visit, and which pages they access most. This data helps you optimise your discovery layer (robots.txt, llms.txt, sitemap) and understand whether your content is being indexed by AI search products like ChatGPT, Claude, and Perplexity. Without tracking, you have no visibility into whether your AI readiness efforts are working.

What are the major AI bot user-agent strings to detect?

The major AI bot user-agents are: GPTBot and ChatGPT-User (OpenAI), ClaudeBot and anthropic-ai (Anthropic), PerplexityBot (Perplexity), Google-Extended and Googlebot (Google), Applebot-Extended (Apple), Bytespider (ByteDance/TikTok), Meta-ExternalAgent and FacebookBot (Meta), Amazonbot (Amazon), cohere-ai (Cohere), CCBot (Common Crawl), Diffbot, and YouBot.

How do I detect bots in Next.js middleware?

In Next.js middleware, access the user-agent from the request headers using req.headers.get("user-agent"). Match it against a map of known bot user-agent strings. When a match is found, send a fire-and-forget POST request to your tracking API route to log the visit, then call NextResponse.next() to allow the bot to access the page normally.

Should I block AI bots that I detect?

In most cases, no. AI bot traffic is beneficial because it leads to your content appearing in AI-generated responses, which drives high-converting referral traffic. Track bots to understand your AI visibility, not to block them. If you need to restrict access, use robots.txt directives, not middleware blocking.

What database schema should I use for bot tracking?

A bot_crawls table with columns for id (uuid), bot_name (text), user_agent (text), path (text), method (text), country (text from Vercel geo headers), and created_at (timestamptz). Add indexes on created_at (DESC) for time-based queries, bot_name for bot-specific filtering, and path for page-level analytics.

How do I build a bot analytics dashboard?

Query your bot_crawls table with SQL aggregations: GROUP BY bot_name for visit counts per bot, GROUP BY path for most-crawled pages, GROUP BY DATE(created_at) for crawl trends over time, and GROUP BY country for geographic distribution. Display the results in an admin dashboard page with tables and charts.

Does bot tracking affect website performance?

Not if implemented correctly. The middleware detection is a simple string comparison that takes microseconds. The logging call to the tracking API is fire-and-forget: the middleware does not wait for the API response before continuing. The bot receives the page at full speed. The tracking happens asynchronously in the background.

How can I use bot tracking data to improve my AI visibility?

Analyse which pages bots visit most and ensure those pages have strong answer capsules, structured data, and llms.txt references. Identify bots that are not visiting and check your robots.txt for accidental blocks. Track crawl frequency trends to measure whether changes to your discovery layer increase bot activity. Compare crawl patterns before and after adding new content to understand what AI systems find most valuable.