CommonMark ipynb export + image attachment embedding

[mmcky]

Summary

Adds myst build --ipynb support that produces notebooks with plain CommonMark markdown cells — compatible with vanilla Jupyter Notebook, JupyterLab (without jupyterlab-myst), and Google Colab. Also adds an option to embed images as base64 cell attachments for fully self-contained notebooks.

---

What's Changed

Bug fixes (built on top of upstream PR jupyter-book#1882)

• Kernelspec from frontmatter — frontmatter parameter was accepted but never used; now populates metadata.kernelspec and metadata.language_info correctly
• Language detection — no longer hardcoded to "python"; derives from frontmatter kernelspec
• Log message — says "Exported IPYNB" instead of "Exported MD"
• +++ markers — stripped globally from markdown cells (was only matching start of string)
• Package homepage — corrected URL from myst-to-md to myst-to-ipynb

Additionally fixes two issues discovered during real-world validation against QuantEcon lectures:

• Epigraph/pull-quote directives silently dropped during export (#7)
• Cross-references with empty URLs in resolved nodes (#8)

CommonMark serialization mode

New markdown: commonmark option in export config triggers an AST pre-transform that converts MyST-specific nodes to CommonMark equivalents before writeMd serialization.

exports:
  - format: ipynb
    markdown: commonmark

18 directive/role mappings implemented:

MyST Node	CommonMark Output
math block	$$...$$
inlineMath	$...$
admonition	> **Title** ... blockquote
exercise	**Exercise N** ...
solution	**Solution** ... (or dropped via dropSolutions)
proof/theorem/lemma	**Theorem N (Title)** ...
tabSet	Bold tab titles + content
container (figure)	 + italic caption
container (table)	Bold caption + GFM table
card / grid / details / aside	Appropriate blockquote/bold fallbacks
mystDirective / mystRole	Unwrapped children
mystTarget / comment	Dropped
code blocks	Stripped MyST-specific attributes
Node identifier/label	Stripped to prevent (id)= prefixes

Key design: uses { type: 'html' } AST nodes for math to prevent mdast-util-to-markdown from escaping LaTeX special characters.

Image attachment embedding

New images: attachment option embeds images as base64 cell attachments for self-contained notebooks.

exports:
  - format: ipynb
    markdown: commonmark
    images: attachment

Two-phase hybrid architecture:

1. Phase 1 (myst-cli): collectImageData() walks AST image nodes, resolves filesystem paths, reads & base64-encodes into Record<url, ImageData>
2. Phase 2 (myst-to-ipynb): embedImagesAsAttachments() post-serialization regex rewrites ![alt](url) → ![alt](attachment:name) with cell attachments field

Validated on 24 QuantEcon lectures: 50 images across 12 notebooks embedded, 0 external references remaining.

Epigraph / pull-quote serialization (fixes #7)

The myst-to-md container handler only handled figure, table, and code kinds — quote containers (produced by {epigraph} and {pull-quote} directives) were silently dropped.

Fix: Added kind === 'quote' branch that:

• Finds the blockquote child and serializes it via the default blockquote handler
• Appends optional caption as > — Attribution line

Cross-reference URL fallback (fixes #8)

When MyST resolves same-page cross-references (e.g., {ref}, {eq}), the addChildrenFromTargetNode transform sets html_id on the node but doesn't always propagate identifier or label. The myst-to-md serializer was only checking urlSource → #label → #identifier, resulting in empty [text]() links.

Fix: Extended the URL fallback chain to: urlSource → #label → #identifier → #html_id → url → ''

Note: A third case was identified where {ref} roles with multi-line bodies fail to parse entirely. This is an upstream parser limitation filed as jupyter-book/mystmd#2724.

Exercise / solution code cell lifting

Exercises and solutions can contain code-cell nodes that should become executable notebook cells. These were being serialized as markdown instead of being lifted to top-level code cells.

Fix: liftCodeCellsFromGatedNodes() pre-processes the AST to extract code cells from exercise/solution containers, splitting surrounding markdown content into separate cells. Handles both direct gated nodes and blocks wrapping gated nodes.

Documentation

• New docs/creating-notebooks.md — comprehensive ipynb export guide
• Updated docs/documents-exports.md — added ipynb to format table
• Updated docs/frontmatter.md — added ipynb to format values
• Updated docs/myst.yml — TOC entry for creating-notebooks
• Updated packages/myst-to-ipynb/README.md — features and usage

---

Test suite — 154 tests passing

myst-to-ipynb (55 tests)

File	Tests	Coverage
basic.yml	13	Core: styles, headings, code, links, images, block markers
frontmatter.yml	4	Kernelspec: Python, Julia, Python3, R
commonmark.yml	18	All directive/role mappings, identifier stripping
attachments.yml	5	Integration: single/multi image, dedup, no-match, no-data
attachments.spec.ts	7	Unit: basename, embed/skip/dedup logic
run.spec.ts	8	Exercise/solution code cell lifting

myst-to-md (99 tests, 6 new)

File	New Tests	Coverage
directives.yml	3	Epigraph, epigraph with attribution, pull-quote
references.yml	3	URL fallback for remote refs, html_id heading, html_id equation

---

Files Changed (36 files, +3122 / -11)

New files

• packages/myst-to-ipynb/src/commonmark.ts — AST pre-transform (~500 lines, 18 node types)
• packages/myst-to-ipynb/src/attachments.ts — Post-serialization image embedding (~100 lines)
• packages/myst-to-ipynb/src/types.ts — Shared ImageData interface
• packages/myst-to-ipynb/tests/commonmark.yml — 18 CommonMark mode tests
• packages/myst-to-ipynb/tests/attachments.yml — 5 attachment integration tests
• packages/myst-to-ipynb/tests/attachments.spec.ts — 7 attachment unit tests
• docs/creating-notebooks.md — New documentation page

Modified files

• packages/myst-to-ipynb/src/index.ts — IpynbOptions, attachment wiring, empty cell filter, exercise lifting
• packages/myst-to-ipynb/tests/run.spec.ts — Test runner supporting frontmatter + options in YAML
• packages/myst-to-ipynb/tests/basic.yml — Expanded to 13 tests
• packages/myst-to-ipynb/tests/frontmatter.yml — 4 kernelspec tests
• packages/myst-to-ipynb/package.json — Fixed homepage URL
• packages/myst-to-ipynb/README.md — Updated documentation
• packages/myst-cli/src/build/ipynb/index.ts — collectImageData(), options passthrough, log fix
• packages/myst-to-md/src/directives.ts — Epigraph/pull-quote container handler
• packages/myst-to-md/src/references.ts — Cross-reference URL fallback chain with html_id
• packages/myst-to-md/tests/directives.yml — 3 epigraph/pull-quote tests
• packages/myst-to-md/tests/references.yml — 3 cross-reference fallback tests
• docs/documents-exports.md — ipynb in format table
• docs/frontmatter.md — ipynb in format values
• docs/myst.yml — TOC entry

---

Tracking issue: QuantEcon/mystmd#2 (full PLAN)
Related: QuantEcon/meta#292 · jupyter-book/mystmd#1882
Real-world validation: QuantEcon/lecture-python-programming.myst#363 — 24 lectures, 0 MyST syntax leaks

---

[Copilot]

Pull request overview

Adds first-class .ipynb export support to the MyST toolchain, including CommonMark-friendly notebook markdown output and optional image embedding via Jupyter cell attachments.

Changes:

• Introduces myst-to-ipynb package to serialize MyST mdast into Jupyter notebook JSON, with optional CommonMark transforms.
• Adds image attachment embedding support (collect image bytes in myst-cli, rewrite image references + add attachments in myst-to-ipynb).
• Wires ipynb into myst-frontmatter export types/validation, myst-cli build/export flows + --ipynb CLI option, and adds docs.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/myst-to-ipynb/tsconfig.json	Adds TS build configuration for the new package.
packages/myst-to-ipynb/package.json	Defines new package metadata, deps, and build/test scripts.
packages/myst-to-ipynb/README.md	Documents notebook export features and frontmatter usage.
packages/myst-to-ipynb/CHANGELOG.md	Adds initial changelog stub for the package.
packages/myst-to-ipynb/.eslintrc.cjs	Enables linting via shared config.
packages/myst-to-ipynb/src/index.ts	Implements mdast → ipynb conversion, CommonMark option, and attachment embedding hook.
packages/myst-to-ipynb/src/commonmark.ts	Adds AST transforms to convert MyST constructs into CommonMark-compatible mdast.
packages/myst-to-ipynb/src/attachments.ts	Adds markdown post-processing to rewrite images into attachment: references + build attachments dict.
packages/myst-to-ipynb/src/types.ts	Defines ImageData shape for attachment embedding.
packages/myst-to-ipynb/tests/run.spec.ts	Snapshot-style YAML-driven test runner for mdast→ipynb serialization.
packages/myst-to-ipynb/tests/basic.yml	Adds coverage for baseline markdown/cell behaviors.
packages/myst-to-ipynb/tests/commonmark.yml	Adds coverage for CommonMark conversion behaviors.
packages/myst-to-ipynb/tests/frontmatter.yml	Adds coverage for kernelspec/frontmatter→notebook metadata.
packages/myst-to-ipynb/tests/attachments.yml	Adds end-to-end coverage for attachment embedding behavior in produced notebooks.
packages/myst-to-ipynb/tests/attachments.spec.ts	Unit tests for embedImagesAsAttachments.
packages/myst-to-ipynb/tests/example.ipynb	Adds an example notebook fixture.
packages/myst-frontmatter/src/exports/types.ts	Adds ipynb to ExportFormats.
packages/myst-frontmatter/src/exports/validators.ts	Recognizes .ipynb extension as ipynb export format.
packages/myst-cli/package.json	Adds dependency on myst-to-ipynb.
packages/myst-cli/src/cli/options.ts	Adds --ipynb CLI option helper.
packages/myst-cli/src/cli/build.ts	Exposes --ipynb on myst build.
packages/myst-cli/src/build/build.ts	Includes ipynb in format selection logic.
packages/myst-cli/src/build/build.spec.ts	Updates format-selection tests to include ipynb.
packages/myst-cli/src/build/utils/collectExportOptions.ts	Allows .ipynb as a valid output extension.
packages/myst-cli/src/build/utils/localArticleExport.ts	Routes ExportFormats.ipynb to the new export runner.
packages/myst-cli/src/build/ipynb/index.ts	Implements runIpynbExport and image-data collection for attachments.
docs/myst.yml	Adds notebooks page to docs navigation.
docs/frontmatter.md	Documents ipynb as a supported export format.
docs/documents-exports.md	Adds ipynb to export overview + CLI examples.
docs/creating-notebooks.md	New documentation page for notebook exporting.
.changeset/witty-tigers-hunt.md	Declares patch releases for affected packages.
.changeset/config.json	Groups versioning for myst-to-ipynb alongside related packages.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.