Coverage check: distinguish intentional curation from accidental gaps

[dacharyc]

Summary

The llms-txt-freshness check (proposed rename to llms-txt-coverage in #16) currently flags any sitemap URL missing from llms.txt as a gap. This works well for detecting drift, but produces false positives when site owners intentionally curate their llms.txt to include only a subset of pages.

For example, a site might exclude directory/index pages that just link to other pages already in llms.txt, or API reference pages that aren't useful in markdown form. The current thresholds (95% pass, 80% warn) penalize this intentional curation.

Inciting issues

• llms-txt-freshness: how should coverage handle intentionally excluded pages? afdocs#46 — Cloudflare contributor hitting this with intentionally curated llms.txt
• llms-txt-freshness seems to fail on coverage not freshness afdocs#17 — suggested measuring "transitive coverage" (pages reachable through linked markdown) rather than just direct listing

Proposed approach: configurable thresholds + exclusion globs

The coverage check serves different use cases that need different behavior. Rather than a separate "mode" toggle, two configuration knobs cover all cases:

1. Configurable thresholds — pass/warn percentages (defaults: 95/80)
2. Exclusion globs — patterns subtracted from the sitemap before calculating coverage

These compose to handle three personas:

Use case	Exclusions	Thresholds	Effect
Full parity (site wants llms.txt to mirror sitemap)	none (default)	95/80 (default)	Current behavior, no config needed
Curated (site intentionally includes a subset, e.g. Stripe at ~16% coverage)	none	0/0	Check still runs and reports coverage %, but never fails
Hybrid (site wants strict coverage but with known exclusions, e.g. Cloudflare)	intentional gaps	95/80	Exclusions shrink the denominator; remaining pages held to strict standard

Example config:

options:
  coveragePassThreshold: 95    # default
  coverageWarnThreshold: 80    # default
  coverageExclude:
    - "/api/reference/**"
    - "/internal/**"

Setting thresholds to 0 effectively makes the check informational without needing a separate mode concept. The check still reports the percentage and lists what's missing, but never warns or fails.

Transitive coverage (longer-term)

Transitive coverage (following links within llms.txt's linked markdown files to count reachable pages) could make the check smarter so that most sites wouldn't need exclusions. However:

• It's unclear how many hops agents realistically follow before giving up and trying other discovery methods. The value of transitive coverage as a metric depends on empirical data about agent behavior.
• Even with 1-hop link following, heavily curated sites like Stripe only reach ~53% of their sitemap (analysis).

This could be explored as a future enhancement, ideally backed by testing around agent link-following behavior, but shouldn't block the threshold + exclusion approach.

Open questions

• Should the spec prescribe these specific configuration options, or just note that implementations should account for intentional curation and leave the mechanism to implementers?
• Are there other signals (beyond exclusion lists) that could help distinguish intentional curation from drift?
• What other considerations should inform this design?

[dacharyc]

Tangentially related: agent-ecosystem/afdocs#38 investigates whether sitemap URLs blocked by robots.txt Disallow directives should be filtered from the URL pool.

This feeds into the same core question here: what should the denominator be when calculating coverage? If a site blocks pages via robots.txt, those pages arguably shouldn't count against coverage either. That's a site-declared exclusion mechanism that already exists, separate from the tool-configured exclusion globs proposed above.

Worth considering as part of the overall design for how the coverage denominator gets constructed.

[njb90]

Firstly, the AF docs spec is a really great resource.

For us, along with markdown content parity, these are intentionally different to provide a more targeted experience for agents.

We generate our llms.txt from our menu file and then have curated set of exclusions (we also have issues around llms.txt length already which we are addressing).

We also have highly customized md files which we sanitize and have certain pages with different content to target agents where the html is not useful for an agent.

For both of these I think the approach should be informational and provide the % without penalizing.

You could argue that these checks are less about agent friendliness and more general maintenance processes. In the case of the md vs html content we think there's scenarios where it's very much beneficial to have very different and very targeted content for agents in the md vs the HTML.

[nicoleomeara-astro]

We intentionally exclude older versions of our on-prem offering, as well as our CLI, from our llms.txt. Excited about this update to the spec!

[devalog]

We (Fern) intentionally exclude changelog entries by default from our own llms.txt as well as llms.txt files for our customers. For our own documentation, changelog files were taking up approximately 80% of the file.

There's an option to override this behavior with a custom file.

[dacharyc]

Ok, spec is updated to reflect the intentional curation case in v0.4.0. I'll follow with an afdocs implementation update soon-ish.

Thank you for weighing in here, everyone!

[dacharyc]

Ok, afdocs v0.17.0 has been released with support for intentional curation of the llms.txt content. 🎉

Thanks everyone for weighing in here!

The details are here: https://afdocs.dev/checks/observability.html#configuring-coverage

I've added a few different escape hatches:

• Configurable warn/fail thresholds: you can set them lower if you know you intentionally exclude some subset of pages, and/or you can set them to 0 and the check becomes informational instead of producing a warn/fail result. This allows you to keep an eye on changes without expecting full parity.
• I've added a --coverage-exclusions <patterns> flag for sites that want to omit certain glob patterns from matching, and have also documented the ones we were already silently ignoring proactively. So if you know you want to omit /changelogs, you can add an exclusion for that.

Hopefully this range of solutions provides something that works for everyone.

I'm going to go ahead and close this issue since I'm the reporter, but if you want to further discuss this, feel free to open a new one and link to here, and I'm happy to consider any further conversation on this.