Write llms.txt and control AI crawlers

What these tools do

AI Crawler Tools covers the two plain text files that decide how AI systems treat a website. llms.txt is an invitation: a short Markdown guide to your most useful pages, written for language models and the tools built on them. robots.txt is the gatekeeper: it tells each crawler which parts of the site it may fetch. The generator writes the first, the builder writes the second, and the checker reads any robots.txt the way a crawler would.

How llms.txt is laid out

The format comes from the llms.txt proposal at llmstxt.org. The file lives at the root of the site, at /llms.txt, and uses a fixed order so it is easy to read by people and by programs:

• A heading with the site or project name. This is the only required part.
• A quoted summary with the key facts a reader needs to understand the rest.
• Optional details: paragraphs or lists, but no headings.
• Sections of links, each under its own second-level heading. Every item is a Markdown link, optionally followed by a colon and a short note.
• An Optional section for secondary links that a reader can skip when space is short.

# Acme Garden Supplies

> Seeds, tools and growing guides for small gardens.

## Guides

- [Starting seeds indoors](https://example.com/guides/seeds.md): Timing by zone

## Optional

- [Company history](https://example.com/about.md)

The proposal also suggests offering a clean Markdown copy of each page at the same address with .md added, and linking to those copies from llms.txt. The generator checks that every link is a full web address and that the heading and summary are in place.

What llms-full.txt is

llms-full.txt is a companion file that many documentation sites publish next to llms.txt. Instead of links, it holds the full text of every page in one file, so a tool can load everything in a single request. It is a common convention rather than part of the proposal itself. Tick the box in the generator to get an outline with a heading and source address for each page, ready for you to paste the content in.

Training, search and user fetchers

AI companies now run several bots each, and they do different jobs. Blocking one does not block the others.

• Training crawlers such as GPTBot, ClaudeBot, CCBot and Meta-ExternalAgent collect pages that may be used to train future models. Blocking them keeps new content out of future training sets. It does not remove anything already collected.
• Control tokens such as Google-Extended and Applebot-Extended are not separate crawlers. They tell Google and Apple whether pages they already crawl for search may be used for AI training. Blocking them does not affect Google Search, Siri or Spotlight.
• Search crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot build the indexes that AI search answers cite and link to. Blocking them can make your site disappear from those answers.
• User-triggered fetchers such as ChatGPT-User and Perplexity-User visit a single page because a person asked about it. Several operators say these fetchers may not follow robots.txt, because a person and not a crawler made the request.

The middle preset, Block training, is the usual choice for sites that want to stay visible in AI search without feeding model training.

How the checker decides

The checker follows the Robots Exclusion Protocol, RFC 9309, the same rules Google documents for its own crawlers:

• A crawler obeys the group whose User-agent line names it, compared without regard to case. If several groups name it, their rules are combined. Only if none does, it falls back to the User-agent: * group.
• Among the rules that match the path, the longest one wins. When an Allow and a Disallow rule are the same length, Allow wins.
• * matches any run of characters, and $ at the end means the path must end there. Disallow: /*.pdf$ blocks every PDF but not /file.pdf?page=2.
• An empty Disallow: blocks nothing, and /robots.txt itself is always allowed.

Privacy

Nothing you type leaves this device. The page has no server behind it: the files are written by your browser and downloaded straight from memory. Your entries are kept in this browser's storage so they are still here next time, and nowhere else. Use Clear in any tool to remove them.

Questions

Does robots.txt actually stop AI crawlers?

The large operators document their bots and say they follow robots.txt, and most do. It is a request, not a lock: a badly behaved scraper can ignore it, and some user-triggered fetchers say openly that they may. To truly keep pages private, put them behind a login.

Will blocking AI crawlers hurt my Google ranking?

No. Google-Extended controls AI training only and Google states it is not a ranking signal. The builder never touches Googlebot or Bingbot, which serve normal search.

Why can't the checker fetch my robots.txt by itself?

Browsers block a page on one site from reading files on another unless that site allows it, which robots.txt files almost never do. Getting around that would mean sending your address through a server, and this tool has none. Copy and paste takes a few seconds.

Should I block the older tokens anthropic-ai and Claude-Web?

They no longer appear in the operator's current documentation, which lists ClaudeBot, Claude-User and Claude-SearchBot. Leaving the older names in a block list does no harm, so the Block training preset keeps them.

Is llms.txt an official standard?

It is a proposal, published in 2024 and adopted by many documentation sites and developer tools. No major AI company has promised to read it, so treat it as a low-cost extra that helps tools which do.

Where do the crawler descriptions come from?

Each one is based on the operator's own published documentation, checked in October 2026. Bots and their names change, so check the operator's page before relying on a single entry.