diff --git a/README.md b/README.md index 56fbabb..f0259e9 100644 --- a/README.md +++ b/README.md @@ -29,11 +29,11 @@ fall back on training data, or silently work with partial information. ## What the Spec Covers -The spec defines **22 checks across 7 categories**: +The spec defines **23 checks across 7 categories**: | Category | Checks | What it evaluates | |----------|--------|-------------------| -| Content Discoverability | 6 | Discovery index exists, is valid, fits in a single fetch, links resolve, links point to markdown, embedded directives pointing agents to `llms.txt` | +| Content Discoverability | 7 | Discovery index exists, is valid, fits in a single fetch, links resolve, links point to markdown, embedded directives in HTML and markdown pointing agents to `llms.txt` | | Markdown Availability | 2 | `.md` URL support, content negotiation via Accept headers | | Page Size | 4 | Rendering strategy (SPA/CSR detection), markdown size, HTML size (pre/post conversion), content start position | | Content Structure | 3 | Tabbed content serialization blowup, section header quality, code fence validity | @@ -55,7 +55,7 @@ If you can only do a few things, these have the highest impact: 3. **Keep pages under 50K characters** of content. Break up mega-pages. ([Why](SPEC.md#category-3-page-size-and-truncation-risk)) 4. **Add an `llms.txt` pointer** to the top of every docs page. - ([Example](SPEC.md#llms-txt-directive)) + ([Example](SPEC.md#llms-txt-directive-html)) 5. **Don't break your URLs.** Use same-host HTTP redirects if you must move content. ([Details](SPEC.md#category-5-url-stability-and-redirects)) 6. **Monitor your agent-facing resources.** Keep `llms.txt` fresh, verify @@ -65,7 +65,7 @@ If you can only do a few things, these have the highest impact: ## Test Your Docs [`afdocs`](https://afdocs.dev) is a companion CLI tool -and Node.js library that implements this spec. It runs all 22 checks against +and Node.js library that implements this spec. It runs all 23 checks against your documentation site and reports what's working, what's not, and what to fix. ```bash diff --git a/SPEC.md b/SPEC.md index 8bea41f..6b55526 100644 --- a/SPEC.md +++ b/SPEC.md @@ -3,8 +3,8 @@ | | | |--------------|--------------------------------------------------------------| | **Status** | Draft | -| **Version** | 0.4.0 | -| **Date** | 2026-04-21 | +| **Version** | 0.5.0 | +| **Date** | 2026-04-25 | | **Author** | Dachary Carey + community contributors | | **URL** | https://agentdocsspec.com | | **Repository** | https://github.com/agent-ecosystem/agent-docs-spec | @@ -16,7 +16,7 @@ Documentation sites are increasingly consumed by coding agents rather than human readers, but most sites are not built for this access pattern. Agents hit truncation limits, get walls of CSS instead of content, can't follow cross-host redirects, and don't know about emerging discovery mechanisms like -`llms.txt`. This spec defines 22 checks across 7 categories that evaluate how +`llms.txt`. This spec defines 23 checks across 7 categories that evaluate how well a documentation site serves agent consumers. It is grounded in empirical observation of real agent workflows and is intended as a shared standard for documentation teams, tool builders, and platform providers. @@ -123,8 +123,9 @@ are ordered by impact based on observed agent behavior: 4. **Put a pointer to your `llms.txt` at the top of every docs page.** A simple blockquote directive that tells agents where to find the documentation index. - Anthropic does this; it works. - Check: `llms-txt-directive` + Anthropic does this; it works. Include it in both the HTML DOM and in + markdown versions of pages. + Checks: `llms-txt-directive-html`, `llms-txt-directive-md` 5. **Don't break your URLs.** If you must move content, use same-host HTTP redirects. Avoid cross-host redirects, JavaScript redirects, and soft 404s. @@ -489,41 +490,82 @@ but would benefit from formal standardization. The proposal's existing "Optional" H2 section could be leveraged for secondary/versioned content, but the nesting pattern goes further by distributing content across multiple files. -### `llms-txt-directive` - -- **What it checks**: Whether documentation pages include a directive, visible - to agents but not necessarily to human readers, pointing to `llms.txt` or - another discovery resource. -- **Why it matters**: Anthropic's Claude Code documentation - (`code.claude.com/docs`, hosted on Mintlify) includes a directive as a - blockquote at the top of every markdown page telling agents to fetch the - documentation index at `llms.txt`. In practice, agents see this directive, - follow it, and use the index to find what they need. It's simple, - low-effort, and observed to work in real agent workflows. This is the - agent equivalent of a "You Are Here" marker. The directive can be visually - hidden (e.g., using a CSS clip-rect technique) as long as it remains in - the DOM and survives HTML-to-markdown conversion. Avoid `display: none`, - which some converters strip. The directive should be present in - server-rendered HTML or in the markdown source; avoid relying solely on - client-side JavaScript injection, since most agents fetch pages without +### `llms-txt-directive-html` + +- **What it checks**: Whether the HTML version of documentation pages includes + a directive, visible to agents but not necessarily to human readers, pointing + to `llms.txt` or another discovery resource. +- **Why it matters**: Agents that fetch rendered HTML pages have no built-in + way to discover that a documentation index exists at `/llms.txt` or that + markdown versions of pages may be available. An in-page directive serves as + an agent "You Are Here" marker that points them to the index. The directive + can be visually hidden (e.g., using a CSS clip-rect technique or `sr-only` + class) as long as it remains in the DOM and survives HTML-to-markdown + conversion. Avoid `display: none`, which some converters strip. The + directive should be present in server-rendered HTML; avoid relying solely + on client-side JavaScript injection, since most agents fetch pages without executing JS. +- **Detection considerations**: Implementations must distinguish intentional + agent-facing directives from incidental mentions of `llms.txt`. Navigation + items (e.g., sidebar links to a page *about* `llms.txt`), JSON-LD metadata, + `