llms.txt and AI crawler access: the 20-minute technical GEO setup

Two plain-text files control most of your technical AI visibility: robots.txt decides which AI crawlers may read your site, and llms.txt tells them what matters once they're in. Auditing the first and writing the second takes about 20 minutes, and it's the highest-leverage technical GEO work there is — a blocked crawler means an engine that can never cite you.

Part 1: Audit robots.txt for AI crawlers

These are the 10 user agents that matter for AI visibility:

The most common failure isn't an explicit block — it's a wildcard User-agent: * / Disallow: / left over from staging, or a security/bot-management plugin that added AI bots to a blocklist in 2023 when that was fashionable. Per RFC 9309, a specific user-agent group overrides the wildcard, so the fix is either removing the block or adding explicit Allow groups for the bots above. (Our free crawler checker reads your file and flags each of the 10 bots individually.)

Part 2: Write a valid llms.txt

llms.txt is a markdown file at your site root that gives AI systems a curated map of your most important content. The format is minimal:

# Acme
> Acme is inventory software for restaurants —
> real-time stock tracking across locations.

## Product
- [Features](https://acme.com/features): what Acme does
- [Pricing](https://acme.com/pricing): plans from $29/mo

## Resources
- [Docs](https://acme.com/docs): setup and API
- [Comparisons](https://acme.com/vs): Acme vs alternatives

• One H1 with your site name — required.
• A blockquote summary right under it — the elevator pitch engines will reuse.
• Sections of markdown links with one-line descriptions. Curate hard: 10–30 links, not your whole sitemap.
• Prioritize the pages you want quoted: pricing, comparisons, docs, methodology.

Part 3: Verify the whole chain

1. Fetch your key pages with JavaScript disabled — is the content there?
2. Confirm robots.txt allows the 10 bots above.
3. Confirm /llms.txt returns the file (correct content type, no redirect to a 404 page).
4. Run a full GEO audit — it checks all of this plus structured data and content shape in one pass.

Then measure whether it moved the needle: technical access is the floor, not the ceiling. The trend you care about is mentions and citations per engine, week over week.