Conversation
Replace ~34 redundant attack-vector-heavy security tests with ~12 focused escaping-coverage tests that verify html.escape() is called at every user-data insertion point. - Replace TestXSSPrevention (12 tests) with TestEscapingCoverage (12 tests) using a single <b>MARKER</b> probe at each insertion point - Collapse TestUltimateEvilAnnData from 7 tests into 1 combined test - Remove 6 CSS injection tests from TestBadColorArrays (covered by test_css_colors_sanitized in TestEscapingCoverage) - Remove XSS tests from TestEvilReadme (covered by test_readme_content_escaped) - Remove TestCSSAttacks and TestEncodingAttacks (redundant) - Remove unused `re` import Test count: ~130 → 108 (all passing, no coverage loss)
Replace ~95 uses of `Any` across 7 files with proper types: - `obj: Any` on formatter methods → `obj: object` - `dict[str, Any]` with known structure → precise union types (e.g., `dict[str, bool | str | None]`, `dict[str, str]`) - Return types like `list[dict[str, Any]]` → `list[dict[str, str | int | tuple[str, ...] | None]]` - Remove unused `from typing import Any` imports
Major CSS overhaul for the HTML repr: - Convert to native CSS nesting with `&` for BEM modifiers/elements, reducing repetition and improving readability (Chrome 120+, Firefox 117+, Safari 17.2+) - Add `[data-theme="dark"]` and `html[data-theme="dark"]` selectors for Furo and sphinx-book-theme dark mode support - Deduplicate dark mode CSS variables via Python string substitution (`/* __DARK_MODE_VARS__ */` placeholder in css.py) — needed because @media queries can't combine with regular selector lists in CSS
- Add `_get_top_level_selectors()` helper to parse CSS at brace depth 0, since native nesting means nested selectors inherit scope from parents - Update CSS scoping tests to only check top-level selectors - Update bare element selector tests to use depth-aware parsing - Filter out vnu "CSS: Parse Error" from strict HTML5 validation since vnu's CSS parser doesn't support native CSS nesting syntax
|
Thanks for the detailed review @flying-sheep, really appreciate you taking the time. Here's where things stand on each point. Happy to discuss any of these further or adjust course if you see it differently. 1. README Rendering: No dependencies addedFully agreed, and already the case. The markdown rendering is a small inline JS parser (180 lines, zero imports). Server-side content is HTML-escaped with 2. Jinja Templating: Not adoptedI gave this serious thought. Jinja's benefit for HTML generation is auto-escaping: The opt-out model is genuinely safer when templates insert plain data into HTML. The problem is that this architecture doesn't work that way. Why auto-escaping doesn't help hereThe repr builds HTML through ~5 layers of composition where components return pre-rendered HTML strings. At each composition boundary (entry row into section, section into page, badge into header, etc.), the inner HTML must be marked
With ~25-30 Could we restructure to avoid
|
Section-level collapse now uses native HTML <details>/<summary> elements instead of JS-driven div toggling. This removes ~60 lines of JS (toggleSection, data-should-collapse setup, section header click/keyboard handlers) and the .anndata-section--collapsed CSS class in favor of the browser's built-in open/close state. - render_section() and render_empty_section(): <div> → <details>, `open` attr for expanded sections, no data-should-collapse - _render_section_header(): <div> → <summary>, fold icon removed - sections.py: unknown/error sections also converted to <details> - CSS: custom ::before triangle on <summary> with rotation for closed state; removed .anndata-section__fold and max-height transition styles - JS search filter: section.classList → section.open property - render_fold_icon(): returns "" (kept for API compat) - Validators: assert collapsed/expanded via <details open> attr Entry-level expandables (nested AnnData, DataFrames) stay JS-based since <details> cannot wrap adjacent <tr> elements in a table.
# Conflicts: # pyproject.toml
…elector Use `.anndata-section > summary` instead of `.anndata-section__header` in CSS, and drop the class from <summary> tags in Python. The semantic tag makes a dedicated class redundant for section headers. Also unify unknown/error sections to use `anndata-section` class (was `anndata-sec`) so they share the same summary styling.
… selectors The media query reflects the OS preference, which can contradict the app theme (e.g., OS dark + Furo light toggle). Attribute selectors like [data-theme="dark"] match the actual app state. This follows xarray's approach (pydata/xarray#6500).
With only one dark mode block remaining (the @media query was removed in the previous commit), the placeholder substitution machinery is no longer needed. Move variables directly into repr.css.
Move all inline styles to CSS rules for consistency: - .anndata-header__filepath, .anndata-spacer, .anndata-categories__dot, .anndata-entry__custom, .anndata-entry--error, .anndata-badge--error Remove redundant STYLE_SECTION_CONTENT/STYLE_SECTION_TABLE constants (already defined in .anndata-section__content/.anndata-section__table). Remove unused CSS: .anndata-search, .anndata-entry__preview--expanded, .anndata-x, .anndata-x__info, .anndata-text--warning, [data-tooltip]. Add missing anndata-section__table class to unknown sections table. Only STYLE_HIDDEN and dynamic background/CSS-variable values remain inline.
… color The .anndata-text--warning rule was incorrectly removed during cleanup but is still applied in registry.py. The .anndata-dtype--ndarray class was defined in constants and used by formatters but never had a CSS rule, falling through unstyled. It now shares the --array color variable.
Keep both copy_on_write_X setting from main and HTML repr settings from html_rep branch.
Two-tier detection: tier 1 uses the canonical has_xp() protocol check from anndata.compat (catches JAX, numpy >=2.0); tier 2 falls back to duck-typing (shape/dtype/ndim) for arrays that don't yet implement the full protocol (PyTorch, TensorFlow). Also uses __array_namespace__() for backend label resolution and updates stale PR scverse#2063 → scverse#2071.
… arrays Device info (cuda:0, tpu:0, GPU:0, etc.) is now shown inline in the type column instead of being hidden in tooltips. Adds visual inspection test 26.
…ce-based coloring
CuPy ≥12 implements the full Array API protocol, so the dedicated formatter
was redundant. ArrayAPIFormatter now handles CuPy arrays (with GPU:{device.id}
for clean labels) and colors all array-api arrays by device type: GPU green,
TPU teal, CPU/other amber — uniformly across backends.
Also removes unused CSS_DTYPE_ARRAY constant and its CSS selector.
|
Hi, thanks for all the work! Quite a bit smaller now, but +22k still fills me with dread. (I know a lot is tests)
Hmm, I’ll take a look, but I’m not sure I want to risk it existing.
I think you missed that you can instead wrap safe markup in
That makes no sense to me. Using a theme setting if we can detect one and defaulting to the OS setting if we can’t is the best we can do, so why not do that?
Huh, didn’t know that’s possible, I recommended nesting mainly for descendant selectors ( I’m pretty happy with the state of the CSS now, but just FYI, there’s options:
But as said, just some pointers, no need to put a lot of work into that, the CSS looks fine!
We could use a
Link for future reference: w3c/css-validator#431
First,
Just make it return a
what’s that, do you have a link? |
…move markdown parser - Add @media (prefers-color-scheme: dark) as OS-level fallback (Tier 1), explicit light selectors (Tier 2) override when app is in light mode, existing dark selectors (Tier 3) unchanged. CSS variables defined once in css.py with placeholder substitution to avoid duplication. - Make TypeFormatter generic via PEP 695 (TypeFormatter[T]). can_format() returns TypeGuard[T], format() receives narrowed type without manual casts. Duck-typed formatters use TypeFormatter[object] with type: ignore. - Remove markdown-parser.js (6.7KB) and markdown.py. README content shown as plain text via <pre> + textContent (XSS-safe). Remove ~110 lines of markdown-specific CSS. - Add w3c/css-validator#431 reference where CSS nesting validation is skipped. - Update visual_inspect_repr_html.py descriptions for plain-text README.
|
@flying-sheep Thanks for the thorough review! Here's what we've addressed and where we landed on the discussion points. Changes madeDark mode CSS. Added TypeGuard for README as plain text. Removed the JavaScript markdown parser (
CSS validator link. Added reference to w3c/css-validator#431 where we skip CSS nesting validation errors. On Jinja / markupsafeThanks for the
Current approach. Explicit I think the current approach is the right fit for this architecture, but happy to discuss further. |
On the PR size (~22K lines)Happy to discuss what could be simplified or split. Here's an honest breakdown of where the lines went. Summary
Tests and test tooling account for 61% of the PR. The implementation itself is ~8,150 lines. Source code breakdownformatters.py (1,172 lines) — 20 type-specific formatters covering ndarray, masked arrays, sparse matrices, backed sparse, DataFrames, Series, categoricals, lazy columns, dask, awkward, array-API/CuPy, nested AnnData, None, bool, int, float, str, dict, color lists, and generic list/tuple. Each formatter is ~50 lines average, with the larger ones (categorical, array-API) handling color swatches, device info, and dtype CSS classes. This is the primary extension point for ecosystem packages. registry.py (1,044 lines) — The plugin system. Bulk comes from: utils.py (790 lines) — Shared helpers: serialization checking via the IO registry, value preview generation (dicts, lists, strings with truncation), color detection and CSS sanitization (whitelist-based, blocks injection), HTML escaping, memory formatting, key validation. The color sanitization alone is ~60 lines because it validates against CSS named colors, hex, rgb(), and hsl() while blocking url(), expression(), and semicolons. html.py (637 lines) — The entry point. Orchestrates header (shape, badges, README icon, search), section rendering loop, footer (version, memory), and wraps everything with scoped CSS/JS. Handles settings capture, container ID generation, and the overall HTML structure. components.py (618 lines) — Reusable UI components: section headers with fold/expand, entry rows with name/type/preview columns, badges, warning icons, copy buttons, search box. These are the building blocks that ecosystem packages can use directly. sections.py (563 lines) — Section renderers for obs/var DataFrames (with column width calculation), mapping sections (obsm, varm, obsp, varp, layers), uns (recursive dict traversal with depth limit), and raw. init.py (468 lines) — Public API with core.py (401 lines) — Shared rendering primitives: format_number (with comma grouping), table rendering for DataFrame expansion, and entry rendering coordination between formatters and HTML output. lazy.py (346 lines) — Lazy AnnData support. Detects lazy mode, reads partial categories from disk without triggering full materialization, determines column dtypes from storage metadata. Wrapped in try/except with graceful fallback. css.py (97 lines) — CSS loader with dark/light variable placeholder substitution (define color blocks once, substitute into both javascript.py (49 lines) — JS loader. Static assetsrepr.css (1,050 lines) — Scoped CSS with native nesting. Covers: layout grid, section headers, entry rows, type column with dtype-specific colors (12 dtype classes), dark mode (three-tier: OS media query, explicit light override, dark theme selectors for Jupyter/Sphinx), README modal, search box, fold/expand animations, badges, warning/error styling, color swatches, copy buttons, scrollable containers. All scoped under repr.js (509 lines) — Fold/expand toggle, search with regex support and toggle buttons, copy-to-clipboard, README modal with keyboard accessibility, wrap-mode toggle for long type strings, ResizeObserver for responsive layout. css_colors.txt (197 lines) — CSS named colors for TestsAverage test is 16 lines. Tests are split by concern:
test_repr_robustness.py (1,493 lines) is the largest because it covers 72 edge cases: escaping at every user-data insertion point (probe-based, not attack-vector-based), unicode handling, crashing objects, circular references, size limits, concurrent access, and error accumulation. These are intentionally thorough because Test infrastructurehtml_validator.py (836 lines) — Regex-based HTML validator with structured assertions ( conftest.py (272 lines) — Shared fixtures: AnnData factories for various configurations, the Visual inspection harnessvisual_inspect_repr_html.py (3,365 lines) — Generates an HTML page with 26+ scenarios for manual review. Not a pytest test. Includes: basic/empty/view AnnData, lazy mode, backed mode, deep nesting, many categories, custom sections (TreeData/MuData/SpatialData mocks), README modal, adversarial data, ecosystem extensibility demos. The HTML template itself is ~2,200 lines (inline CSS for the test page layout, accordion sections, checklists). This could live in a separate repo or as a notebook, but having it adjacent to the code makes it easy to regenerate during development. What could be reduced?Genuinely open to suggestions. Some candidates:
None of these would change the order of magnitude. The feature has genuine breadth: 20 type formatters, a plugin registry, 11 configurable settings, dark mode, lazy mode support, serialization warnings, and search. For comparison, pandas' The test-to-code ratio of 1.7:1 reflects a deliberate choice: |
The expanded raw subsection now displays index previews matching the main AnnData header, with graceful "not available" fallback when indices are absent or inaccessible.
Upstream added `size: int` to `SupportsArrayApi`, causing `has_xp()` to reject the mock and `coerce_array` to raise.
4 participants
Rich HTML representation for AnnData
Summary
Implements rich HTML representation (
_repr_html_) for AnnData objects in Jupyter notebooks. Builds on previous draft PRs (#784, #694, #521, #346) with a complete, production-ready implementation.Live Demo | Reviewer's Guide (technical details, design decisions, extensibility examples)
Screenshot
Features
Interactive Display
.rawsection showing unprocessed data (Reportn_varsof.rawin__repr__#349)Visual Indicators
unspalettes (e.g.,cell_type_colors)unsvaluesuns["README"])Serialization Warnings
Proactively warns about data that won't serialize:
/(deprecated)Compatibility
.anndata-reprprevents style conflictsread_lazy()(categories, colors)Extensibility
Three extension mechanisms for ecosystem packages (MuData, SpatialData, TreeData):
obst/vart,mod)See the Reviewer's Guide for examples and API documentation.
Testing
python tests/visual_inspect_repr_html.pyRelated
sparse_datasetby removingscipyinheritance #1927 (sparse scipy changes), feat: array-api compatibility #2063 (Array-API)Acknowledgments
Thanks to @selmanozleyen (#784), @gtca (#694), @VolkerH (#521), @ivirshup (#346, #675), and @Zethson (#675) for prior work and discussions.
Technical Notes and Edits
Lazy Loading
Constants are in
_repr_constants.py(outside_repr/) to prevent loading ~6K lines onimport anndata. The full module loads only when_repr_html_()is called.Config Changes
pyproject.toml: Addedvartto codespell ignore list (TreeData section name).Edit (Dec 27, 2024)
To simplify review and reduce the diff, I've merged settylab/anndata#3 into this PR. That PR was originally created as a follow-up to explore additional features based on the discussion with @Zethson about SpatialData/MuData extensibility.
What changed:
.rawsection - Expandable row showing unprocessed data (Reportn_varsof.rawin__repr__#349)Edit (Jan 4, 2025)
Moved detailed implementation documentation (architecture, design decisions, extensibility examples, configuration reference) to the Reviewer's Guide to keep this PR description focused on features.
Code refactoring:
html.pyinto focused modules for maintainabilitycomponents.py(badges, buttons, icons)sections.py(obs/var, mapping, uns, raw)core.py(avoids circular imports)utils.pyFormatterContextconsolidates all 6 rendering settings (read once at entry, propagated via context)html.pyreduced from ~2100 to ~740 lines, clean import hierarchyNew features:
read_lazy()AnnData objects (experimental) - indicates when obs/var are xarray-backed(lazy)indicator on columnsBug fixes:
adata-text-mutedclass for uniform appearanceRelated issue discovered:
read_lazy()returns index values as byte-representation strings (e.g.,"b'cell_0'"instead of"cell_0") - seeISSUE_READ_LAZY_INDEX.mdEdit (Jan 6, 2025)
Smart partial loading for
read_lazy()AnnData:Previously, lazy AnnData showed no category previews to avoid disk I/O. Now we do minimal, configurable loading to get richer visualization cheaply: only the first N category labels and their colors are read from storage (not the full column data). New setting
repr_html_max_lazy_categories(default: 100, set to 0 for metadata-only mode).Visual tests reorganized: 8 (Dask), 8b (lazy categories), 8c (metadata-only), 9 (backed).
Edit (Jan 6, 2025 - continued)
FormattedOutput API and architecture:
Clean separation between formatters and renderers - formatters inspect data and produce complete
FormattedOutput, renderers only receiveFormattedOutput(never the original data).The
FormattedOutputdataclass fields were renamed to be self-documenting:meta_contentpreview(text) orpreview_html(HTML)html_content+is_expandable=Trueexpanded_htmlhtml_content+is_expandable=Falsepreview_htmlis_expandableexpanded_html is not Nonetype_htmltype_namevisually)Naming convention:
*_htmlsuffix indicates raw HTML (caller responsible for escaping), plain text fields are auto-escaped.UI/UX improvements:
▼/▲arrows instead of⋯/▲for consistencyEdit (Jan 7, 2025)
Test architecture overhaul:
Tests reorganized from a single file into 10 focused modules for maintainability and parallel execution:
test_repr_core.pytest_repr_sections.pytest_repr_formatters.pytest_repr_ui.pytest_repr_warnings.pytest_repr_registry.pytest_repr_lazy.pytest_html_validator.pyHTMLValidator class (
conftest.py) provides structured HTML assertions:Key features: regex-based (no dependencies), section-aware matching, exact attribute matching to avoid "obs" matching "obsm".
Optional strict validation when dependencies available:
validate_html5()- W3C HTML5 + ARIA (requiresvnu)validate_js()- JavaScript syntax (requiresesprima)Jupyter Notebook/Lab compatibility tests (13 new tests in
TestJupyterNotebookCompatibility):Validates CSS scoping, JavaScript isolation, unique IDs across multiple cells, and Jupyter dark mode support.
Bug fix:
readme-modal-titleID is now unique per container to prevent ID collisions when multiple AnnData objects are displayed in the same notebook.Edit (Jan 8, 2025)
Maintainability improvements:
_render_entry_rowandrender_formatted_entryto eliminate duplicationget_formatter_for()andlist_formatters()methods to FormatterRegistry__init__.pystatic/directorytests/repr/html_validator.pymodule (conftest.py: 960→270 lines)_repr_constants.pyrender_entry_type_cell()signaturelazy.pymodulestatic/css_colors.txtfor easy updatesFile structure changes:
API simplifications:
render_entry_type_cell()now acceptsTypeCellConfigdataclass instead of 10 individual parametersis_lazy_adata(),is_lazy_column(),get_lazy_categories(),get_lazy_categorical_info()importlib.resources.files()(Python 3.9+)Edit (Jan 9, 2025)
Robustness & escaping coverage testing:
Added 108 tests in
test_repr_robustness.pyacross 14 test classes:html.escape()is called at every user-data insertion point using a<b>MARKER</b>probe__repr__,__len__,__sizeof__, properties)Escaping tests trust
html.escape()(stdlib) and only verify it's called at every insertion point, rather than exercising the escaping mechanism itself with attack vectors.Test cleanup:
Removed redundant and overly-specific tests to focus on meaningful coverage. Tests now verify behavior that matters (e.g., XSS escaped, errors visible, truncation applied) rather than testing identical code paths multiple times.
Visual inspection: Consolidated to 26 scenarios with single comprehensive "Evil AnnData" test combining all adversarial patterns.
Fixes:
repr_html_max_readme_sizeto_settings.pyitype stubspytest.warnsfor expected warnings)Updated stats:
Edit (Jan 16, 2025)
Error handling consolidation:
Refactored error handling to use a single
errorfield inFormattedOutputinstead of separateis_hard_errorparameters scattered across the codebase.Key changes:
FormattedOutputerror: str | Nonefield with documented precedence overpreview/preview_htmlFallbackFormatterFormatterRegistry.format_value()render_formatted_entry()is_hard_errorparam, now detects viaoutput.error_validate_key_and_collect_warnings()(key_warnings, is_key_not_serializable)- key issues mark as not serializable, preserving previewError vs Warning separation:
output.error: Hard rendering failure - row highlighted red, error message replaces previewoutput.is_serializable=False: Serialization warning - red background, but preview preservedNew behavior when formatters fail:
This prevents long error messages from appearing in HTML while preserving full details in warnings for debugging. Serialization issues (like non-string keys, lambdas, custom objects) preserve the value preview while showing the reason in the tooltip.
Updated stats:
Edit (Jan 26, 2025)
Review response changes (addressing @flying-sheep's review):
Typing:
Any→objectReplaced all ~95 uses of
Anyacross 7 files. Formatter method signatures now useobj: objectsince AnnData'sunsaccepts genuinely arbitrary objects and formatters handle AnnData-like objects (e.g., MuData) via duck typing.dict[str, Any]with known structure replaced with precise union types.CSS: Native nesting + dark mode + variable dedup
repr.cssto native CSS nesting (&). Selector repetitions of.anndata-reprreduced from 173 to 13. File length unchanged (~1164 lines) because the feature surface is genuinely large (~68 component blocks, 14 dtype colors, copy button, README styling, state variants), not because of repetition.[data-theme="dark"]for Furo/sphinx-book-theme) alongside existing Jupyter/VS Code detection.@media (prefers-color-scheme: dark)block and theme-selector block.&--variant) produce invalid CSS at nesting depth 2+ (browser treats&as:is(parent child), so&--viewbecomes:is(.anndata-repr .anndata-badge)--view). 7 modifier rules flattened to sibling selectors.Security tests simplified
Replaced ~34 attack-vector-heavy tests with 12 focused escaping-coverage tests. Each test puts a
<b>MARKER</b>probe at one user-data insertion point and verifies it appears escaped. RemovedTestCSSAttacks,TestEncodingAttacks; trimmedTestBadColorArrays,TestEvilReadme; consolidatedTestUltimateEvilAnnDatato 1 test. Total: 108 tests (14 classes), down from 123 (16 classes).Other:
FormatterContext.column_namerenamed toFormatterContext.keyFormatterRegistry.format_value()Future-Proofing: Related PRs and Issues
This PR includes explicit handling and/or code references to track compatibility with several in-progress or future changes. The following PRs/issues may trigger updates to the
_reprmodule:Already Handled
_reprSparseMatrixFormatteruses duck typing fallbackformatters.py:242,260,307ArrayAPIFormattervia duck typingformatters.py:771,1135ArrayAPIFormatterMay Require Updates When Merged
LazyCategoricalDtypeAPICategoricalArrayinternalslazy.py(all functions)obsformatters.py:159Recommended Post-Merge Actions
When feat: add
LazyCategoricalDtypefor lazy categorical columns #2288 merges:CategoricalFormatterandlazy.pyto use the newLazyCategoricalDtypeAPIget_lazy_categorical_info()extracts category count by manually navigatingobj.variable._data.array— replace withdtype.n_categoriesanddtype.head_categories(n)isinstance(dtype, LazyCategoricalDtype)for cleaner detectionWhen Add support for lists in obs #1923 is resolved:
_check_series_serializability()informatters.pyto recognize list-of-strings as serializableWhen feat: allow gpu io in
sparse_datasetby removingscipyinheritance #1927 merges:SparseMatrixFormatterstill works with new sparse array classesis_sparse()utility or the new classes have a stable API, the duck typing incan_format()(checking fornnz,tocsr,tocsc) could be simplified to direct type checksWhen feat: array-api compatibility #2063/feat: support array-api #2071 stabilize:
ArrayAPIFormatterduck typing (shape/dtype/ndim) follows the Array API standard and is the correct approachis_array_api_compatible(), could use that instead of manual attribute checks"cubed": "Cubed"toknown_backendsdict inArrayAPIFormatterfor prettier display labelsInternal API Usage Inventory
Current patterns accessing internal/private APIs that may be replaceable:
lazy.py:_get_categorical_array()col.variable._data.arrayisinstance(dtype, LazyCategoricalDtype)lazy.py:get_lazy_category_count()CategoricalArray._categories["values"].shape[0]dtype.n_categorieslazy.py:get_lazy_categorical_info()._categories,._ordereddtype.n_categories,dtype.orderedlazy.py:get_lazy_categories()read_elem_partial()on private._categoriesdtype.head_categories(n)lazy.py:is_lazy_adata()obs.__class__.__name__ == "Dataset2D"SparseMatrixFormatter.can_format()nnz,tocsr,tocscArrayAPIFormatter.can_format()shape,dtype,ndimBackedSparseDatasetFormatter.can_format()formatattr