llms.txt vs robots.txt

Both files sit at the root of your domain and both speak to crawlers — but they answer opposite questions. robots.txt is a gatekeeper; llms.txt is a tour guide. robots.txt defines who may enter and what they may read. llms.txt describes what the site is about so AI agents know which pages deserve their attention.

What does robots.txt control?

robots.txt is a decades-old standard enforced by all major crawlers. It uses User-agent and Allow/Disallow directives to tell crawlers which paths are off-limits. For AI, a typical setup allows search-bot crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) and blocks training-only crawlers (Bytespider, CCBot). Getting this wrong has real consequences: if Cloudflare's "Block AI Crawlers" toggle is on, it injects Disallow rules that silently remove your site from ChatGPT and Perplexity search results.

What does llms.txt provide?

llms.txt is a proposed convention (Answer.AI, 2024) with no enforcement mechanism: nothing crawls it automatically, but AI tools that support the convention check for it at the site root. It is markdown — an H1, a summary blockquote, H2 sections of links with descriptions, and a Related section. It answers "what is this site, and what should I read first?" rather than "which paths may I access?"

What is the key difference?

Permissions versus orientation. robots.txt is declarative policy — it restricts access and has been adopted by every crawler. llms.txt is descriptive content — it recommends structure and is early-stage adoption. A site that allows AI crawlers but has no llms.txt is crawlable but opaque; a site with llms.txt but blocked robots.txt is invisible. They are complementary: publish both, and let llms.txt's Related section point to your sitemap.

Which AI bots should you allow?

Allow the crawlers that power AI search: GPTBot and OAI-SearchBot (ChatGPT), ClaudeBot and anthropic-ai (Claude), PerplexityBot, and Google-Extended (AI Overviews). Decide separately about training-only crawlers like Bytespider and CCBot — many sites block them because they consume bandwidth without sending search traffic. The generator's robots.txt check shows the status of nine AI crawlers for any domain you paste.

Do you need both files?

Yes. robots.txt is non-negotiable for AI visibility — without Allow rules, no AI search engine sees your site regardless of llms.txt. llms.txt is the cheap, future-proofing extra: minutes to create with the generator, harmless while adoption is early, and a first-mover advantage if the convention becomes standard. See also llms.txt vs sitemap.xml.