feat: add hidden OCR template context for post-processing

[evrenesat]

Summary

This PR introduces an experimental OCR context path for post-processing prompts across supported desktop platforms.

When the selected post-processing prompt contains ${OCR} or ${ocr}, Handy captures text from the frontmost app context (OCR) and injects it into the prompt template before sending to the post-processing model.

This is intentionally a hidden/experimental feature for now:

• No UI changes
• No new settings UI
• No public frontend command/API changes

Note: If guided, I can make follow-up changes needed for merge readiness (including UI exposure if desired).

Intent

The goal is to improve post-processing quality by adding optional visual context from the active window/screen, especially for fixing terminology and typo-like dictation misunderstandings.

I’m using the following addition to the default prompt and getting better results than non-OCR post-processing:

Relevant context (OCR’ed from active window, use for fixing typos and terminology):
${OCR}

I intentionally kept scope minimal and hidden to validate usefulness first, without increasing UI complexity or feature surface.

Architecture / Flow

• Existing backend/frontend architecture remains unchanged (Rust backend in src-tauri, React/TS frontend in src).
• Prompt template expansion in src-tauri/src/actions.rs now supports:
  • ${output} (existing transcript variable)
  • ${OCR} / ${ocr} (new OCR context variable)
• OCR is attempted only when:
  1. OCR variable exists in the selected prompt
  2. Existing experimental_enabled setting is true
• OCR text is truncated to 8000 chars to limit prompt growth.
• On OCR failure, missing permission, or unsupported conditions, OCR resolves to an empty string and processing continues safely.

Platform Notes

macOS

• Added:
  • src-tauri/swift/macos_ocr.swift
  • src-tauri/swift/macos_ocr_bridge.h
  • src-tauri/src/macos_ocr.rs
• Updated:
  • src-tauri/build.rs (bridge build/link wiring)
  • src-tauri/src/lib.rs (module registration, macOS-gated)
  • src-tauri/src/actions.rs (template expansion + OCR integration + tests)
• Frameworks linked:
  • Foundation
  • AppKit
  • CoreGraphics
  • Vision
• Permission behavior:
  • Screen recording permission requested on demand
  • Request-at-most-once per app run
  • Graceful fallback to empty OCR context if denied/unavailable

Windows

• Added Windows OCR provider integration for ${OCR} / ${ocr} in post-processing flow.
• Uses src-tauri/src/windows_ocr.rs for frontmost-window OCR capture on Windows.
• Wired into shared prompt-template expansion in src-tauri/src/actions.rs.
• Behavior is fail-safe: OCR errors resolve to empty context and do not break transcription/post-processing flow.

Linux

• Added:
  • src-tauri/src/linux_ocr.rs
• Linux OCR strategy:
  • X11 session: active-window X11 capture path (with root-window fallback), then Tesseract.
  • Wayland session: screenshot-tool fallback, then Tesseract.
• Wayland fallback tools:
  • GNOME preferred: gnome-screenshot -f <temp.png>
  • KDE preferred: spectacle -b -n -o <temp.png>
  • NOTE: Unlike other implementations, these cli tools take full-screen screenshots.
• Backend selection is detected once at app startup and cached.
• Packaging/build updates:
  • Install tesseract-ocr in Ubuntu CI jobs (.github/workflows/build.yml)
  • Add Debian runtime dependency tesseract-ocr (src-tauri/tauri.conf.json)
  • Add Linux x11 crate dependency in src-tauri/Cargo.toml
• Current limitation:
  • Wayland path depends on screenshot tools being installed.
  • This is an interim pragmatic implementation; portal/D-Bus integration can improve this later.
  • GNOME’s org.gnome.Shell.Screenshot D-Bus path is noted as a potential follow-up for better window-targeted behavior.

Cross-platform Safety

• OCR providers are OS-gated.
• Existing transcription and post-processing flows are preserved.
• Unsupported/failing OCR paths degrade gracefully to empty OCR context.

Prior Art Check

I searched upstream issues/PRs for OCR-related work and did not find a matching implementation.

Testing

• Added/updated unit tests in src-tauri/src/actions.rs for OCR template resolution and truncation behavior.
• Added Linux OCR unit tests in src-tauri/src/linux_ocr.rs for backend selection and parsing logic.
• Frontend build and tests pass locally.
• Linux build/test and AppImage build were verified in Ubuntu VM.
• Note: in CommandLineTools-only macOS environments, Apple Intelligence @Generable macro tooling may fail independently; this PR does not change Apple Intelligence behavior.

AI Assistance Disclosure

• AI used: Yes
• Tools used: GPT-5 Codex
• Usage: Extensive assistance across implementation, debugging, build validation, and documentation

[cjpais]

Hey, thanks for this. it's gonna take me a while to get to and merge this. Right now I'm mainly trying to fix stability issues.

I can't say 100% this will be merged, but I do think that this is a good thing. However, it would be best if we can do this on multiple platforms.

[evrenesat]

it would be best if we can do this on multiple platforms.

Sure, but I don't have access to neither Linux or Windows. And Mac version uses completely built-in libraries of the operation system, so it doesn't bring any more dependencies in that sense. And being it's a hidden feature, it wouldn't cause any confusion for users of other platforms, so in it's current form, waiting for feature parity seems a bit unnecessary?

Still, with this addition Handy become perfect for me, both feature and reliability wise, so I'm fine living with my own build :)

Thanks for the sharing and maintaining Handy! Let me know if you want me to improve this PR.

[cjpais]

Understand, and it will probably be merged without other support, just ideally like to not feature things off just to one platform where possible

[cjpais]

Just dropping a note, this is going to take me a while to merge I think. I need to keep the feature set stable right now. I feel I'm drowning under PR's and issues and starting to get burned out.

[evrenesat]

@cjpais I have added implementation for all platforms in this PR now. Linux one is a bit hacky, but for now that's the only solution I could come up with.