Scan a website

Guide

llms.txt, llms-full.txt, llms-all.txt and AGENTS.md Explained

These AI text files are often confused. Learn which are website guides, which are documentation dumps, and which belong in a code repository.

Several new text files have appeared as websites and development tools adapt to AI.

The most frequently discussed are:

These names are often grouped together, but they do not perform the same function. Some are proposed website discovery formats. Others are conventions created by documentation platforms. AGENTS.md is mainly intended for AI coding agents working inside a software repository.

This guide explains what each file does, which ones are standardized and how to write them correctly.

The quick comparison

File Main purpose Typical location Status
llms.txt Curated guide to important website content /llms.txt Proposed open format
llms-full.txt Consolidated full-text version of documentation /llms-full.txt Common convention, not a universal standard
llms-all.txt Varies by implementation Custom No widely accepted specification
AGENTS.md Instructions for AI coding agents Repository root or subdirectory Open coding-agent convention

The most important distinction is simple:

llms.txt and llms-full.txt are intended to expose website or documentation content. AGENTS.md tells coding agents how to work on a codebase.

What is llms.txt?

llms.txt is a proposed Markdown file placed at the root of a website:

https://example.com/llms.txt

Its purpose is to give language-model tools a concise overview of the website and a curated list of useful resources.

The original proposal describes it as a way to provide LLM-friendly information for inference-time use. It is designed to reduce the noise created by navigation, advertising, complex layouts and JavaScript-heavy pages.

A useful comparison is:

This comparison is helpful, but llms.txt does not have the same level of standardization or crawler support as robots.txt and XML sitemaps.

Is llms.txt an official web standard?

Not in the same sense as robots.txt, HTTP or XML sitemaps.

It is an open proposal published in 2024. Several documentation platforms and software projects have adopted it, but support is not universal.

Google states that it does not use llms.txt as a special signal for its search index or generative search features. Google may crawl it as an ordinary text file, but creating one does not improve or reduce Google rankings.

This does not make the file useless. It means its value depends on whether a particular tool, agent or retrieval system chooses to read it.

Use llms.txt as an optional compatibility feature, not as proof that a website is fully AI-ready.

The recommended llms.txt structure

The proposed format uses Markdown and follows a predictable order.

A basic file looks like this:

# Example Company

> Example Company provides website auditing tools for developers,
> website owners and SEO professionals.

Use these resources to understand the product, its checks and its limitations.

## Product

- [AI Readiness Checker](https://example.com/): Check whether a website is accessible and understandable to AI systems.
- [Features](https://example.com/features): Review the technical checks performed by the platform.
- [Pricing](https://example.com/pricing): Compare available plans and usage limits.

## Documentation

- [Getting Started](https://example.com/docs/getting-started): Run your first website analysis.
- [Scoring Method](https://example.com/docs/scoring): Understand how the readiness score is calculated.
- [Crawler Checks](https://example.com/docs/crawlers): Learn how crawler access is tested.

## Policies

- [Privacy Policy](https://example.com/privacy): How submitted website data is handled.
- [Terms](https://example.com/terms): Terms governing use of the service.

## Optional

- [Company Story](https://example.com/about): Background information about the company.
- [Blog](https://example.com/blog): Articles about AI search and machine-readable websites.

The proposed specification uses:

  1. An H1 containing the website or project name
  2. An optional blockquote summary
  3. Optional explanatory text
  4. H2 sections
  5. Markdown links with optional descriptions

A section named Optional has a specific proposed meaning. Tools can omit those resources when they need a smaller context.

How to write a good llms.txt file

Keep it curated

Do not copy your entire sitemap into llms.txt.

Select the pages that best explain:

A smaller, accurate file is more useful than a large list of low-value URLs.

Use absolute URLs

Prefer:

- [Documentation](https://example.com/docs): Product documentation.

Instead of:

- [Documentation](/docs): Product documentation.

Absolute URLs remain clear when the file is copied, downloaded or processed outside your website.

Describe every important link

A title alone may not explain why a page matters.

Weak:

- [API](https://example.com/api)

Better:

- [API Documentation](https://example.com/api): Authentication, endpoints, rate limits and response examples.

Descriptions should be factual and concise.

Link to canonical pages

Do not include:

Each resource should lead directly to the preferred public version.

Keep claims consistent with the website

Do not use llms.txt to introduce claims that do not appear on the linked pages.

The file should summarize and organize public information. It should not become a hidden alternative version of your marketing copy.

Update it when important content changes

Review the file when you:

An outdated machine-readable file can create more confusion than no file at all.

What is llms-full.txt?

llms-full.txt is commonly used as a single, consolidated text version of a documentation website.

Instead of listing links, it may contain the actual content from many documentation pages in one file.

Mintlify describes its implementation as a file that combines an entire documentation site into a single context source for AI tools.

A simplified example could look like this:

# Example Product Documentation

## Introduction

Example Product checks whether websites can be crawled and understood by AI systems.

## Getting Started

1. Enter a public website URL.
2. Start the analysis.
3. Review crawler, metadata and content-structure checks.
4. Fix failed checks.
5. Run the analysis again.

## Crawler Access

The crawler check reviews robots.txt rules, response codes and public accessibility.

## Structured Data

The structured-data check identifies valid JSON-LD and relevant schema types.

When should you create llms-full.txt?

It is most useful for:

It may be less useful for:

A full-document file can become extremely large. Large files may exceed practical context limits, waste processing resources or mix relevant information with unrelated material.

For a large site, topic-specific Markdown exports may be more useful than one enormous file.

Is llms-full.txt part of the original specification?

The original llms.txt proposal focuses mainly on the curated index file and Markdown versions of linked pages. It also describes generated context files such as llms-ctx.txt and llms-ctx-full.txt in one implementation.

The exact name llms-full.txt became popular through documentation platforms and tooling. It is now a recognizable convention, but its format is not enforced by one universal standards body.

This means you should document what your version contains.

For example:

# Example Documentation

> This file contains the complete public documentation for Example Product.
> Last generated: 2026-07-18.
> Private, deprecated and duplicate pages are excluded.

What is llms-all.txt?

llms-all.txt does not have a widely accepted specification.

Different tools may use the name to mean:

Because the meaning is unclear, do not assume that an AI crawler will search for or understand it.

Use llms-all.txt only when:

For general public implementation, llms-full.txt is the more recognizable filename for consolidated documentation.

Do not publish both files with identical content unless there is a concrete compatibility reason. Duplicate machine-readable files increase maintenance work and create uncertainty about which version is authoritative.

What is AGENTS.md?

AGENTS.md is a Markdown file containing instructions for AI coding agents.

It is often described as a README for agents. It gives tools predictable access to information that helps them understand, test and modify a software project.

A typical file may contain:

Example:

# AGENTS.md

## Project overview

This repository contains the CrawlMirror website and auditing service.

## Setup

- Install dependencies: `npm install`
- Start development server: `npm run dev`
- Create a production build: `npm run build`

## Tests

- Run unit tests: `npm test`
- Run type checking: `npm run typecheck`
- Run linting: `npm run lint`

## Code style

- Use TypeScript strict mode.
- Use semantic HTML for public pages.
- Do not add dependencies without a clear need.
- Keep crawler checks separate from presentation logic.

## Security

- Never commit API keys.
- Do not log submitted authentication headers.
- Treat all submitted URLs as untrusted input.
- Preserve SSRF protection in URL-fetching code.

## Before completing a task

- Run tests.
- Run the production build.
- Explain any checks that could not be completed.

Does AGENTS.md make a public website more visible to AI?

Usually, no.

AGENTS.md is primarily read by coding agents operating on a repository or local project. It is not a general replacement for llms.txt, structured data, HTML content or an XML sitemap.

OpenAI Codex, for example, reads AGENTS.md files to obtain project-specific instructions before performing development work.

A public AGENTS.md file may be reachable on a website, but that does not mean search or answer engines will treat it as a visibility signal.

Use it to improve AI-assisted development, not as an AEO tactic.

Where should AGENTS.md be placed?

Place a general file at the repository root:

/project
  AGENTS.md
  package.json
  src/
  tests/

Large repositories can also contain more specific files in subdirectories:

/project
  AGENTS.md
  frontend/
    AGENTS.md
  backend/
    AGENTS.md

A root file can define general rules. A nested file can explain requirements for a specific part of the project.

Keep instructions relevant to the directory in which they apply.

Common mistakes with AI-readable files

Treating proposals as guaranteed ranking factors

There is no evidence that simply adding these files produces search rankings or AI citations.

Generating files and never updating them

Machine-readable content must remain consistent with the website and codebase.

Exposing private information

Do not include:

Assume that every publicly accessible file can be copied and archived.

Creating an enormous llms.txt file

llms.txt should normally be a guide, not a complete website dump.

Use llms-full.txt or separate Markdown resources when full content is genuinely useful.

Using AGENTS.md for marketing

Coding instructions should be precise and operational. Promotional language wastes context and can confuse development agents.

Forgetting the website itself

These files do not compensate for inaccessible HTML, broken links, blocked crawlers or weak documentation.

Recommended implementation

For most business websites:

  1. Maintain strong HTML pages.
  2. Publish an accurate XML sitemap.
  3. Configure robots.txt deliberately.
  4. Add relevant structured data.
  5. Create llms.txt as an optional curated guide.
  6. Skip llms-full.txt unless you have substantial documentation.
  7. Skip llms-all.txt unless a specific system requires it.
  8. Add AGENTS.md to the code repository when AI coding tools are used.
  9. Test every public URL included in the files.
  10. Review the files after major website changes.

Frequently asked questions

Should every website have llms.txt?

No. It is optional. It is most useful when a website has clear documentation, authoritative resources or structured product information that can be curated.

Does Google use llms.txt?

Google says it does not use llms.txt as a special signal for Google Search or its generative search features.

Should I use llms-full.txt or llms-all.txt?

Use llms-full.txt when you need a consolidated version of public documentation. Use llms-all.txt only when a particular integration defines and requires it.

Can llms-full.txt contain my entire website?

It can, but that is rarely the best approach for a large site. Exclude duplicate, outdated, private and low-value content.

Is agents.md case-sensitive?

The established convention uses the uppercase filename AGENTS.md. Follow the expected casing so compatible coding tools can locate it reliably.

Can I use both llms.txt and AGENTS.md?

Yes. They solve different problems. llms.txt can guide systems toward public website resources. AGENTS.md guides coding agents working on the repository.

Test your AI-readable files

Publishing a file is only the first step.

CrawlMirror can check whether your website exposes the files correctly, whether linked resources are reachable and whether more fundamental crawlability problems are preventing AI systems from accessing your content. Validate your file with the llms.txt checker.

Related guides: What is Answer Engine Optimization? · How to make a website readable for AI

Sources and further reading

Validate your llms.txt

← All guides