How to Create an llms.txt File for Your Website

Q: What is the difference between llms.txt and robots.txt?

robots.txt controls access, it tells crawlers which pages they can visit. llms.txt provides context, it tells AI systems what your site is about and how to use it. They serve complementary purposes. A site should have both.

While robots.txt tells AI crawlers what they can access and sitemaps help them discover your pages, neither file actually explains what your website does or how to use it. That is the gap llms.txt fills.

llms.txt is a plain text file that gives large language models a concise, structured overview of your website. Think of it as a README for AI. When an AI agent encounters your domain for the first time, llms.txt is the fastest way for it to understand your business, your products, your API capabilities, and how to recommend you to users.

The standard was proposed by Jeremy Howard (co-founder of fast.ai) and has gained rapid adoption across the AI ecosystem. Thousands of websites now serve an llms.txt file, and an increasing number of AI tools and agents actively check for it. The specification is documented at llmstxt.org.

This guide covers the complete format specification, real production examples, best practices, and step-by-step instructions for creating your own llms.txt file. Use the AgentReady scanner to check if your site already has one configured.

Why llms.txt Matters

The comprehension problem

AI crawlers can access your pages, but that does not mean they understand your site. A language model crawling a typical website encounters navigation menus, cookie banners, marketing copy, footer links, and dozens of pages that may or may not be relevant to a user's query. The model has to piece together what your business actually does from scattered fragments of HTML.

llms.txt solves this by providing a single, authoritative source of truth. Instead of the AI system inferring what you do from scattered page content, you explicitly tell it: "Here is what we do, here is what we offer, here is how to use our tools, and here are the most important pages."

Accuracy in AI responses

When an AI system understands your business clearly, it describes and recommends your product more accurately. Without llms.txt, an AI might describe your SaaS product as "a website about accounting software" when it is actually "an AI-powered bookkeeping platform for UK freelancers." The difference in positioning matters when an AI is recommending solutions to users.

AI agent interoperability

As AI agents become more autonomous, they need to quickly understand what a website offers and whether it is relevant to their current task. llms.txt serves as a discovery mechanism for agents, similar to how robots.txt serves as a permissions mechanism for crawlers. An agent can read llms.txt in seconds and decide whether to explore further or move on.

Competitive advantage

Most websites do not have an llms.txt file yet. Creating one gives your site an immediate advantage when AI systems are deciding which sources to cite or recommend. It is a low-effort, high-impact addition to your AI readiness stack.

The llms.txt Format Specification

The llms.txt format uses Markdown-style headers and sections to organise information. It is designed to be human-readable while also being structured enough for AI systems to parse reliably.

Core structure

An llms.txt file follows this structure:

# Site Name

> One-line description of what this site/product does.

## About

A paragraph explaining the product, who it is for, and what problems it solves.

## Features

- Feature 1: Brief description
- Feature 2: Brief description
- Feature 3: Brief description

## API / Tools

- [Tool Name](https://yourdomain.com/api/endpoint): What it does
- [MCP Endpoint](https://yourdomain.com/api/mcp): Model Context Protocol server

## Key Pages

- [Pricing](https://yourdomain.com/pricing): Plans and pricing information
- [Documentation](https://yourdomain.com/docs): Full product documentation
- [Blog](https://yourdomain.com/blog): Latest articles and updates

## Contact

- Email: hello@yourdomain.com
- Support: support@yourdomain.com

Section-by-section breakdown

Title (required)

The first line should be a Markdown H1 heading with your site or product name. This is the primary identifier for AI systems.

# p0stman

Tagline (recommended)

A blockquote immediately after the title providing a one-line description. Keep it under 100 characters. This is the single most important line in the file -- it is what AI systems use to quickly categorise your site.

> AI-native product studio building voice agents, websites, and apps.

About section (required)

A 2-4 sentence paragraph explaining what the product does, who it is for, and what makes it different. Write this as if you are explaining your product to a knowledgeable AI assistant who needs to recommend it to a user.

Features section (recommended)

A bulleted list of key features or capabilities. Keep each item to one line. Focus on what differentiates your product, not generic features that every competitor also has.

API / Tools section (recommended for products with APIs)

List any API endpoints, MCP tools, or programmatic capabilities. Use Markdown link format [Name](URL): Description. This section is especially important for AI agents that might want to interact with your product programmatically.

Key Pages section (recommended)

Links to the most important pages on your site. This helps AI systems find the most relevant content without crawling your entire site. Include pages like pricing, documentation, about, blog, and key feature pages.

Contact section (optional)

Contact information for the business. Useful for AI agents that need to recommend how a user can get in touch.

Real Production Examples

Example: SaaS product

# AccountsOS

> AI-powered bookkeeping for UK freelancers and sole traders.

## About

AccountsOS automates bookkeeping, tax calculations, and HMRC submissions
for UK freelancers. It connects to your bank account, categorises transactions
using AI, and prepares your Self Assessment tax return automatically. Built
for non-accountants who want to stay compliant without the complexity.

## Features

- Automatic bank transaction categorisation via AI
- Real-time tax liability calculation (Income Tax + National Insurance)
- One-click Self Assessment submission to HMRC
- Receipt scanning and matching
- MTD-compatible VAT returns
- Accountant collaboration portal

## Pricing

- Free: Up to 50 transactions/month
- Pro: 9.99 GBP/month, unlimited transactions
- Business: 24.99 GBP/month, multi-entity support

## API

- [MCP Server](https://accounts-os.com/api/mcp): Query financial data via MCP
- [REST API](https://accounts-os.com/docs/api): Full API documentation

## Key Pages

- [Pricing](https://accounts-os.com/pricing): Plans and pricing
- [Features](https://accounts-os.com/features): Full feature list
- [Self Assessment Guide](https://accounts-os.com/self-assessment): Tax filing guide
- [Blog](https://accounts-os.com/blog): Latest updates

## Contact

- Email: hello@accounts-os.com
- Support: support@accounts-os.com

Example: Agency / studio

# p0stman

> AI-native product studio. We build voice agents, websites, and apps.

## About

p0stman is a digital product studio specialising in AI-powered applications.
We build custom AI agents, voice interfaces, web applications, and mobile apps
for businesses. Led by Paul Gosnell with 20 years of product experience. We
work as a fractional product team, embedding with clients on 12-month retainers
or delivering fixed-scope projects.

## Services

- AI Agent Development: Custom voice and chat agents using Gemini, Claude, GPT
- Web Application Development: Next.js, React, full-stack applications
- Mobile App Development: Cross-platform mobile applications
- Fractional CPO: Ongoing product leadership and strategy
- AI Readiness Audits: AgentReady scanner for AI visibility

## Tools

- [AgentReady Scanner](https://p0stman.com/agentready): Free AI readiness audit
- [MCP Server](https://p0stman.com/api/mcp): Query services and case studies
- [A2A Agent](https://p0stman.com/api/agent): Agent-to-agent communication

## Key Pages

- [Case Studies](https://p0stman.com/case-studies): Client work and results
- [Services](https://p0stman.com/services): Full service list
- [Process](https://p0stman.com/process): How we work
- [Guides](https://p0stman.com/guides): AI and development guides
- [Contact](https://p0stman.com/contact): Get in touch

## Contact

- Email: hello@p0stman.com
- Twitter: @zerop0stman

Example: Documentation site

# Supabase

> Open-source Firebase alternative with Postgres, Auth, Storage, and Realtime.

## About

Supabase is an open-source backend-as-a-service built on PostgreSQL. It provides
authentication, real-time subscriptions, storage, edge functions, and a REST/GraphQL
API layer. Used by over 1 million developers for building web and mobile applications.

## Features

- PostgreSQL database with row-level security
- Authentication (email, OAuth, magic links, SSO)
- Real-time subscriptions via WebSockets
- Object storage with CDN
- Edge Functions (Deno runtime)
- Auto-generated REST and GraphQL APIs
- Database migrations and branching

## Documentation

- [Getting Started](https://supabase.com/docs/guides/getting-started)
- [JavaScript Client](https://supabase.com/docs/reference/javascript)
- [Python Client](https://supabase.com/docs/reference/python)
- [Auth Guide](https://supabase.com/docs/guides/auth)
- [Database Guide](https://supabase.com/docs/guides/database)

## Pricing

- Free: 2 projects, 500MB database, 1GB storage
- Pro: $25/month, 8GB database, 100GB storage
- Team: $599/month, dedicated support
- Enterprise: Custom pricing

## Contact

- Support: support@supabase.io
- GitHub: github.com/supabase/supabase

Best Practices

Keep it under 100 lines

llms.txt should be concise. AI agents scan this file quickly to decide whether to explore your site further. If the file is too long, the agent may truncate it or skip sections. Aim for 50-100 lines total. If you need to provide more detail, use llms-full.txt.

Lead with the most important information

Put the tagline, about section, and key features at the top. Many AI systems only read the first portion of a file. The first 30% of content is cited 44% of the time by language models, and this applies to llms.txt as well.

Use specific, factual language

Avoid marketing superlatives like "the best" or "world-class." AI systems are trained to recognise and discount marketing language. Instead, use specific, factual descriptions: "AI-powered bookkeeping for UK freelancers" is more useful to an AI than "the ultimate finance solution."

Include pricing if applicable

Users frequently ask AI systems about pricing. If your pricing is public, include it in llms.txt. This ensures AI responses about your product include accurate pricing information rather than guesses or outdated data.

Link to key pages with descriptive text

Use the Markdown link format [Descriptive Name](URL): What this page contains. The descriptive text helps AI systems understand what the linked page contains without needing to crawl it.

Keep it updated

Update llms.txt whenever you add new features, change pricing, or launch new products. An outdated llms.txt is worse than none at all because it feeds inaccurate information to AI systems. Add it to your deployment checklist alongside robots.txt and sitemap updates.

llms.txt vs. llms-full.txt

The llms.txt specification supports an optional companion file called llms-full.txt. The relationship is straightforward:

llms.txt (required): A concise overview, under 100 lines, under 4,000 tokens. This is what agents read first.
llms-full.txt (optional): An extended version with complete documentation, API schemas, detailed feature descriptions, and more. This is for agents with large context windows that want the full picture.

Reference llms-full.txt from your llms.txt file:

# Your Product

> One-line description.

For full documentation optimised for LLMs, see: https://yourdomain.com/llms-full.txt

## About
...

Implementation in Next.js

Static file approach

The simplest approach is to create public/llms.txt in your Next.js project. This file will be served automatically at /llms.txt.

# Create the file
touch public/llms.txt

# Edit with your content
# The file should be plain text with Markdown formatting

Dynamic API route approach

If your llms.txt content needs to be generated dynamically (e.g., to include current pricing from a database), you can create an API route:

// app/llms.txt/route.ts
import { NextResponse } from 'next/server'

export async function GET() {
  const content = `# Your Product

> One-line description.

## About

Your product description here.

## Features

- Feature 1
- Feature 2
- Feature 3

## Key Pages

- [Home](https://yourdomain.com)
- [Pricing](https://yourdomain.com/pricing)
- [Docs](https://yourdomain.com/docs)

## Contact

- Email: hello@yourdomain.com
`

  return new NextResponse(content, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=86400',
    },
  })
}

Middleware considerations

If your Next.js middleware intercepts requests, make sure /llms.txt is excluded from any auth checks or redirects. Add it to your public path bypass list alongside robots.txt and sitemap.xml.

// middleware.ts
export function middleware(req: NextRequest) {
  const publicPaths = [
    '/llms.txt',
    '/robots.txt',
    '/sitemap.xml',
    '/.well-known/agent.json',
    '/mcp.json',
  ]

  if (publicPaths.some(p => req.nextUrl.pathname === p)) {
    return NextResponse.next()
  }

  // ... rest of your middleware
}

llms.txt in the AI Readiness Stack

llms.txt is one component of a comprehensive AI readiness strategy. Here is how it fits with the other pieces:

File	Purpose	Analogy
robots.txt	Controls crawler access	The bouncer at the door
sitemap.xml	Lists all pages for discovery	The building directory
llms.txt	Explains what the site does	The receptionist who gives you context
mcp.json / /api/mcp	Enables programmatic interaction	The API handbook
.well-known/agent.json	Agent-to-agent discovery	The business card for other agents

For a complete AI readiness audit covering all these components, run the AgentReady scanner on your domain.

Common Mistakes

Too long

An llms.txt file that runs to 500 lines defeats the purpose. Agents need a quick overview, not an encyclopedia. Keep it under 100 lines. Use llms-full.txt for extended content.

Marketing copy instead of facts

Phrases like "revolutionary platform" and "best-in-class solution" are noise to an AI system. Use factual descriptions: what the product does, who it is for, what it costs, and how to use it.

Missing links

Always use full URLs in your links, not relative paths. AI systems processing your llms.txt may not have context about your domain. Use https://yourdomain.com/pricing not /pricing.

Outdated information

If your llms.txt says you have three pricing tiers but you actually have four, AI systems will give incorrect recommendations. Treat llms.txt as a living document that gets updated alongside your product.

No llms.txt at all

The biggest mistake is not having one. Even a minimal 20-line llms.txt file is significantly better than nothing. AI systems will attempt to understand your site from HTML content alone, which is slower and less accurate.

Frequently Asked Questions

What is llms.txt?

llms.txt is a plain text file placed at the root of your website (at /llms.txt) that provides a structured summary of your site specifically for large language models. It tells AI systems what your site does, what content and tools are available, and how to interact with your product programmatically. Think of it as a README file for AI.

Where should llms.txt be placed?

llms.txt should be placed at the root of your domain, accessible at https://yourdomain.com/llms.txt. For Next.js projects, place it in the public/ directory. Some sites also offer llms-full.txt with extended content for models with larger context windows.

Is llms.txt an official standard?

llms.txt is an emerging community standard proposed by Jeremy Howard (fast.ai) and documented at llmstxt.org. It is not yet an IETF RFC or W3C standard, but it has gained significant adoption across the AI industry and is recognised by major AI tools and agent frameworks.

How long should llms.txt be?

Keep llms.txt under 100 lines and under 4,000 tokens. AI agents scan this file quickly, so brevity is key. If you need to provide more detail, create a separate llms-full.txt file and link to it from llms.txt. The concise version is always read first.

What is the difference between llms.txt and robots.txt?

robots.txt controls access -- it tells crawlers which pages they can visit. llms.txt provides context -- it tells AI systems what your site is about and how to use it. They serve complementary purposes. A site should have both for comprehensive AI readiness.

Do AI crawlers actually read llms.txt?

Yes. An increasing number of AI tools and agents check for llms.txt when encountering a new website. Claude, ChatGPT plugins, Perplexity, and various AI coding assistants are known to check for this file. The standard is gaining adoption rapidly as AI agent ecosystems mature.

Should I include API documentation in llms.txt?

Include a brief summary of API capabilities and link to full documentation. Do not put your entire API reference in llms.txt. The file should be a concise overview that helps an AI system understand what is available, with links to deeper resources for agents that need the full specification.

How does llms.txt relate to MCP and agent discovery?

llms.txt provides human-readable context for AI systems. MCP (Model Context Protocol) provides machine-readable tool definitions via mcp.json and /api/mcp. They complement each other: llms.txt helps an AI understand what your site does, while MCP enables it to take action. Use both for maximum AI readiness.