Investigate robots.txt Disallow filtering for sitemap-discovered URLs

[dacharyc]

Context

When afdocs discovers page URLs via sitemap (either as the primary source or as a fallback to supplement a thin llms.txt), it does not check whether those URLs are blocked by robots.txt Disallow directives. This matters for agent discoverability: if a page is disallowed in robots.txt, an LLM that respected robots.txt during training would not have crawled it, so the agent has no way to discover the page from its base training data unless it's also listed in llms.txt.

Current behavior

discoverSitemapUrls() reads robots.txt only to find Sitemap: directives. It does not parse Disallow: rules. All sitemap URLs that pass origin and path-prefix filtering are included in the URL pool regardless of robots.txt crawl rules.

Questions to investigate

1. How common is this? Do real documentation sites block doc pages via robots.txt Disallow while still listing them in sitemaps?
2. Scope of parsing needed: robots.txt Disallow rules can include wildcards, user-agent matching, and other complexity. What's the minimum viable parser for this use case?
3. Interaction with llms.txt: If a page is in llms.txt but blocked by robots.txt, should it still be tested? (Likely yes, since the site owner explicitly offered it via llms.txt.)
4. Behavior options: Should blocked URLs be excluded from the pool, or included but annotated in the report?

Related

• Came up during Fall back to secondary discovery sources when llms.txt has fewer URLs than --max-links #27 (discovery source fallback) implementation
• Applies to all sitemap-based discovery, not just the llms.txt+sitemap merge case

[mvvmm]

Upon some further research here, it seems to me that if a site has disallowed routes in it's robots.txt, it would be contradictory for that site to include those routes in its sitemap. That is to say, disallowed robots.txt routes should not exist in sitemap.xml. This Yoast article describes an error in google search console around this issue.

And I think by extension, and for the same reasons, we can also say that it would be wrong for llms.txt files to include routes disallowed by robots.txt.

In my opinion, I see two paths forward for afdocs and the spec, either:

1. Continue to ignore robots.txt when discovering the URL pool (and continue to rely on llms.txt/sitemap), and sites failure to correctly exclude disallowed routes across llms.txt/sitemap will cause some inconsistent afdocs reporting (correctly so?) across checks like llms-txt-freshness
2. Do 1, and implement an additional check that tests for this behaviour specifically, and fails/warns when sites do not correctly exclude disallowed routes in their sitemap/llms.txt

Let me know what you think

[dacharyc]

Yeah, I think I agree with your conclusions here. I'm going to continue to ignore robots.txt for now - I don't necessarily want to go through the effort of testing for this and failing/warning when sites do not exclude disallowed routes. That feels like scope creep for afdocs as it stands currently. I'll revisit if I get feedback later that this is confusing or people want that type of functionality in the tool.