feat: add default support for sphinx-llm

[jahn-junior]

• Have you updated CHANGELOG.md with relevant non-documentation file changes?
• Have you updated the documentation for this change?

---

Add default support for the sphinx-llm extension.

With this configuration, the only command that runs the Markdown builder is make html, as users will need some way to preview these docs before pushing changes to RTD. On my machine, this added between 6 and 15 seconds to the build time depending on project size.

This implementation prevents sphinx-llm from generating Markdown artifacts in the live preview with the new SPHINX_AUTOBUILD_OPTS variable. While it doesn't take long to build the additional files, users won't be happy when that additional time compounds every time the live preview is rebuilt.

sphinx-llm only runs on the html and dirhtml builders, so we don't have to worry about the linkcheck builder.

The Markdown artifacts are still built when running pa11y. However, shaving a few seconds off an uncommon check that takes several minutes isn't worth the complexity it'd bring to the Makefile.

[jahn-junior]

@rkratky What are your thoughts on llms_txt_description?

Do we have any reason to believe it makes a meaningful difference? If so, I'd like to have guidance ready for users on what to write before we add it to the default config.

If we can't yet vouch for its efficacy, then I'd rather leave it out for now to keep this as light as possible.

[medubelko]

Not blocking, but I'm documenting that there's some noise in the output:

<a id="contribute"></a>

# Contribute

* [Test the Ulwazi theme](test-ulwazi-theme.md)

<a id="sphinx-documentation-starter-pack"></a>
<!-- SHORTCUTS -->

The first link is from our convention to add a label to the page title, but I don't know where the last link is from. SHORTCUTS is from the end of links.txt.

[medubelko]

Tests fine on my end. The HTML builder generates the files, while the autobuild and linkcheck builders don't.

Possibly blockers in llms.txt. We have 8 links with cropped optional text. In a few cases it's causing broken links:

- [Style guide linting](reference/automatic_checks_styleguide/index.html.md): The starter pack includes a method to run the [Vale](https://vale.sh/) documentation linter configur...
- [MyST syntax](reference/myst-syntax/index.html.md): The documentation files use a mixture of [Markdown](https://commonmark.org/) and [MyST](https://myst...
[...]
- [Link check](reference/automatic_checks_links/index.html.md): The link check uses Sphinx to access the links in the documentation output and validate whether they...
[...]
- [reStructuredText syntax](reference/rst-syntax/index.html.md): The documentation files use [reStructuredText](https://www.sphinx-doc.org/en/master/usage/restructur...
- [Inclusive language check](reference/automatic_checks_inclusivelanguage/index.html.md): The inclusive language check uses [Vale](https://vale.sh/) to check for violations of inclusive lang...
- [Automatic checks](reference/automatic_checks/index.html.md): The starter pack comes with several automatic checks that you can (and should!) run on your document...
- [Accessibility check](reference/automatic_checks_accessibility/index.html.md): The accessibility check uses [Pa11y](https://pa11y.org/) to check for accessibility issues in the do...
[...]
- [How to contribute](how-to/contributing-myst/index.html.md): We believe that everyone has something valuable to contribute, whether you’re a coder, a writer, or ...

IMO we should go ahead and add meta descriptions to these pages.

[rkratky]

@rkratky What are your thoughts on llms_txt_description?

Do we have any reason to believe it makes a meaningful difference? If so, I'd like to have guidance ready for users on what to write before we add it to the default config.

The general consensus seems to be that it's useful. It's not required but recommended. The guidance could be as simple as "one or two-sentence summary of the docs set."

If we can't yet vouch for its efficacy, then I'd rather leave it out for now to keep this as light as possible.

We can't really vouch for much of anything in this regard :) What is clear is that the description definitely aids when llms.txt is used on-demand (i.e. by an agent through an MCP server, for example). Then it helps the agent orient itself in the library of docs it has at its disposal.

[rkratky]

Not blocking, but I'm documenting that there's some noise in the output:

<a id="contribute"></a>

The first link is from our convention to add a label to the page title

This is unfortunate, but outside the control of the sphinx-llm extension. The extension uses the sphinx-markdown-builder extension under the hood to build the Markdown files (not the main llms.txt file but all the content files). And this is the builder's standard behavior (which is understandable).

I fixed the same behavior in sphinx-llm, so that this doesn't happen in the llms.txt files. But unless we want to do some post-processing on the Markdown files, I think we need to live with it. I don't think it's worth the hassle.

[medubelko]

I don't think it's worth the hassle.

Agreed. I'd rather fix our theme templates so it doesn't happen in the first place.

[rkratky]

Possibly blockers in llms.txt. We have 8 links with cropped optional text.

Cropped text is OK -- that's expected behavior. The extension either uses the html_meta description for each page (if available), or it falls back to the first 100 characters of the first content paragraph on the page.

In a few cases it's causing broken links:

However, the cropping of links is bad -- that's probably a bug that should be fixed. The extension should either strip the markup and leave only the link text, or include the whole thing, so that links wouldn't be mangled like this.

But it's a rabbit hole... no markup should be broken up -- not just links. Stripping all markup would be a tall order. I'm not sure how to handle this gracefully without having to introduce a whole lot of extra logic.