Kitty graphics protocol support

[dsturnbull]

This PR adds kitty graphics protocol support to the graphics branch, building on top of the existing sixel infrastructure. The key insight driving the implementation is that GraphicData is protocol-agnostic — once kitty's APC payload is decoded into a pixel buffer, the entire GPU pipeline (texture upload, cell attachment, shader rendering) works identically to sixel. No renderer changes were needed.

Method

Analyse kovid's spec, wezterm, ghostty, and alacritty-graphics. Turns out not a lot, mostly encoding/decoding machinery, which I generated (see docs)

Architecture

                       ┌──── DCS 'q' ──► sixel::Parser ──┐
 PTY ──► vte-graphics ─┤                                  ├──► GraphicData
                       └──── APC 'G' ──► kitty::Parser ──┘         │
                                 │                                 ▼
                                 │ (responses)              insert_graphic()
                                 ▼                                 │
                           PTY write-back                          ▼
                           ESC _Gi=N;OK                    Grid cells get
                                                           GraphicCell refs
                                                                   │
                                                                   ▼
           display::draw() ──► take_queues() ──► GPU upload ──► shader

The new code lives entirely in alacritty_terminal. The rendering side (alacritty/src/display/) is untouched — kitty images flow through the same insert_graphic() → GraphicCell → texture upload → fragment shader path that sixel already uses.

What's implemented

APC routing — vte-graphics is patched (dsturnbull/vte@kitty) to dispatch apc_start/apc_put/apc_end, following the same pattern as the existing DCS hooks. The VTE state machine already recognised the APC_STRING state; it just wasn't wired to callbacks.

Key-value parser (kitty_parser.rs, ~800 LOC) — Streaming parser for the ESC _G<key>=<value>[,...];<payload>ESC \ format. Handles all documented protocol fields including the overloaded animation keys (c=/r=/z= mean different things in frame context vs display context).

Transmission mediums (decode.rs, ~1100 LOC):

• t=d Direct — inline base64, with lenient padding handling for clients like chafa that independently encode each chunk
• t=f File — base64-encoded file path with offset/size support
• t=t Temp file — relative to $TMPDIR, auto-deleted after read, path traversal validation rejects ../ and absolute paths
• t=s Shared memory — POSIX shm_open + mmap, shm_unlink after read (unix-gated)

Decode pipeline: base64 → zlib (optional, o=z) → format dispatch (RGB f=24, RGBA f=32, PNG f=100) → GraphicData. Chunked transfers (m=0/m=1) decode each chunk's base64 on arrival rather than concatenating base64 strings, which handles the chafa per-chunk-padding case correctly.

Image storage (state.rs, ~1060 LOC) — HashMap<u32, KittyImage> with 320 MiB quota and LRU eviction (matching kitty/ghostty/wezterm). Image number (I=) → image ID (i=) mapping. All 11 delete target pairs:

Target	Pairs	Description
All	d=a/A	Clear all images
By ID	d=i/I	Delete by image ID
By number	d=n/N	Delete by image number
Cursor	d=c/C	Delete at cursor position
Column	d=q/Q	Delete intersecting column
Row	d=r/R	Delete intersecting row
Cell	d=x/X	Delete at specific cell
Cell+Z	d=y/Y	Delete at cell with z-index match
Z-index	d=z/Z	Delete by z-index
Placement	d=p/P	Delete by placement ID
Anim frame	d=f/F	Delete animation frames

The IncludingScrollback (uppercase) variants clean up orphaned images from storage.

Placement (placement.rs, ~600 LOC) — Source rectangle cropping (x=/y=/w=/h=), cell-based scaling (c=/r= via bilinear resize), sub-cell pixel offsets (X=/Y= via transparent padding), cursor movement control (C=1). Pipeline: crop → scale → offset → insert_graphic.

Animation (animation.rs, ~1120 LOC) — Frame storage with lazy single→multi-frame promotion (following the WezTerm pattern). a=f loads frames with base-frame copy and positional blitting. a=c composites between frames (alpha blending and overwrite modes). a=a controls playback (start/stop/loading, loop count). Note: the animation tick loop (advancing frames on a timer and re-uploading textures) is not yet wired into the display event loop — frames are stored and composed but not auto-advanced.

Protocol responses (response.rs) — ESC _Gi=<id>;OK ESC \ / ESC _Gi=<id>;EINVAL:<msg> ESC \ via Event::PtyWrite. Respects q=1 (suppress OK) and q=2 (suppress all). No response when i=0 and I=0 (matching kitty's finish_command_response behaviour).

Testing

184 #[test] functions across the modules:

• 13 end-to-end integration tests that feed raw APC bytes through the full VTE → Parser → Term → dispatch pipeline and verify state
• Unit tests across all streams (decode, state, placement, animation, response, parser)
• scripts/test_kitty_graphics.py — 25 manual protocol-level tests with visual ✓/✗ feedback (query, PNG, chunked, file/tempfile/shm mediums, delete modes, scaling, pixel offsets, animation frames)
• scripts/demo_kitty_graphics.sh — real-world validation with timg, chafa, and mpv

What works end-to-end

• timg -p kitty image.png ✓
• timg -p kitty animation.gif ✓
• chafa -f kitty image.png ✓
• kitty +kitten icat image.png ✓ (chunked PNG transfer)
• Direct RGBA/RGB transmission ✓
• File and shared memory transmission ✓
• Placement, cropping, scaling, pixel offsets ✓
• Delete by all target types ✓

New dependencies

Crate	Purpose
flate2 1.1	zlib decompression (o=z)
png 0.18	PNG decoding (f=100) — also fixes a breaking API change in the window icon loader
image 0.25 (minimal, png+rayon features)	Bilinear scaling for c=/r= cell-based resize
tempfile 3 (dev)	Integration tests for temp file medium

Segfaults with mpv video playback — looking for input

mpv --vo=kitty and timg -V (video mode) send images at 30+ fps, effectively stress-testing the graphics pipeline. Under this load, both tools reliably trigger a segfault in mpv. n.b. this also happens in ghostty!

[wkr111]

🎉I can't believe you did something so amazing!!!🎉🥳