Add semantic token LSP support for Quarto files

[juliasilge]

Addresses #420

What is semantic highlighting, you ask? Why, it is special, extra highlighting that some LSPs provide (as opposed to grammar-provided syntax highlighting):
https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide

After working on this, I understand why we didn't add it a while ago; there is a LOT of bookkeeping here! 😅

The best way to see how this is behaving is to test this in VS Code (not Positron) using Pylance. We do intend to make changes in Positron so it works similarly, but we are a bit in flux right now with our Python LSP.

If you have some Python code that gets semantic tokens highlighted in a regular .py file:

We should mostly get the same semantic tokens highlighted in a .qmd file:

Only some themes support the semantic token highlighting, so be sure to use one of those (like the main built-in themes).

[Copilot]

Pull request overview

This PR adds semantic token support from LSP servers to Quarto documents. Semantic tokens provide enhanced syntax highlighting based on language server analysis rather than grammar-based highlighting. The implementation includes middleware to intercept semantic token requests, convert them to virtual document coordinates, remap token indices between different legend formats, and adjust positions back to real document coordinates.

Key changes:

• Added semantic token provider middleware that creates virtual documents and delegates to embedded language servers
• Implemented token encoding/decoding and legend remapping utilities to handle differences between language server token legends
• Added comprehensive test coverage for token manipulation functions

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
apps/vscode/src/vdoc/vdoc.ts	Added semantic token coordinate adjustment function and registered "semanticTokens" as a virtual document action
apps/vscode/src/test/semanticTokens.test.ts	Added comprehensive test suite for semantic token encoding, decoding, and legend remapping
apps/vscode/src/providers/semantic-tokens.ts	Implemented semantic token provider with legend remapping and coordinate adjustment
apps/vscode/src/lsp/client.ts	Registered semantic token middleware provider in LSP client
apps/quarto-utils/src/semantic-tokens-legend.ts	Defined standard semantic token legend for Quarto documents
apps/quarto-utils/src/index.ts	Exported semantic token legend for use across packages
apps/lsp/src/middleware.ts	Added semantic token capability and handler to LSP middleware

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.