Headless CLI client for YAOS

[enieuwy]

Use case

I run a headless Linux server (no GUI, no Obsidian desktop) that holds my vault files. I want this server to participate in YAOS sync with Obsidian on macOS/iOS. The server would read/write the local filesystem and sync with the YAOS CRDT like any other peer.

This is distinct from the existing "external edit" feature (which watches a filesystem where Obsidian is running). This is YAOS sync to a bare filesystem on a machine without Obsidian installed.

Relation to Issue #2 (Self-hosted runtime)

I see #2 is extracting a runtime-agnostic core layer for server self-hosting. A headless client would be a natural symmetric counterpart: a client peer that participates in the YAOS sync without the Obsidian UI, reading/writing a local filesystem directory.

Where #2 is about "run the YAOS server on my own machine," this is about "run a YAOS client on my server that syncs to a room hosted on Cloudflare (or elsewhere)."

What I'm asking

Would you accept a PR that adds a headless CLI client to YAOS?

The scope would be:

• A Node.js CLI that connects to an existing YAOS room (not a self-hosted server replacement)
• Uses the same VaultSync + diskMirror logic, but with Node fs instead of Obsidian's Vault API
• Shares the CRDT schema, migration logic, and auth flow with the plugin
• Published as @yaos/cli or similar from the same repo

Technical feasibility (from my analysis)

The Obsidian coupling in diskMirror.ts appears mechanical, not architectural:

• File operations: vault.read() → fs.readFile(), vault.modify() → fs.writeFile(), etc.
• Path normalization: normalizePath() → path.normalize() or path.posix.normalize()
• Event watching: vault.on('modify') → chokidar or fs.watch
• ~31 Obsidian API refs, concentrated in ~6 operations

vaultSync.ts is already mostly runtime-agnostic (only normalizePath from Obsidian).

The risk I see: schema drift. YAOS is at schema v2 with active migration logic. A standalone implementation would need to track schema changes reactively. Maintaining this in-repo would leverage the existing test infrastructure and migration paths.

Questions for maintainers

1. Is headless client support in scope for YAOS, or is the project intentionally Obsidian-only?
2. Would you prefer this in the main repo (as a packages/cli workspace) or as a separate repo under the YAOS org?
3. Should this wait for Issue Add a self-hosted Docker runtime #2 to land (runtime-agnostic core extraction), or could it proceed in parallel?
4. Are there architectural invariants I'm missing that would make this harder than "replace VFS calls with fs calls"?

If the answer is "yes, but later" or "yes, with changes," I'm happy to wait or revise the approach. If the answer is "no, out of scope," I'll respect that and build standalone (with the understanding that I'll be tracking an undocumented evolving protocol).

---

Checklist:

• Searched existing issues for "headless", "cli", "terminal", "standalone", "server" — no matches
• Read the self-host RFC (Add a self-hosted Docker runtime #2)
• Reviewed diskMirror.ts and vaultSync.ts coupling

[kavinsood]

Hi, thank you for the writeup and the draft PR! The timing on this is great, with Obsidian CLI recently validating the usecase.

I had considered building a headless client before, but I had rejected it because:

(1) listening to filesystem events in a cross-platform binary is hard. we're probably underestimating obsidian's abstractions on the file API.
(2) i'd have to fight developer licenses on signing executables, which i dont want to.

But adding a headless client is a big add-on for power users, and we can feasibly build this if we constraint to Linux users only. I'm fully onboard with this. We can proceed with the monorepo workspace (@yaos/cli).

I have some hard boundaries to ensure this goes smoothly:

1. We need to first extract the core sync engine into a VaultFS interface (The first half of issue #2). We are not maintaining two parallel monoliths.

2. Atomic writes are non negotiable. I looked at your PR, before this can be merged, all file writes in node must be strictly atomic (writing to .filename.md.tmp, calling fs.fsync, and then executing fs.renameSync).

3. In your PR, there is no local persistence layer. If the CLI is being used as a daemon, then this is fine, but it's a bad idea with cronjobs. Users will blow through their Cloudflare limits. We should do two things:

• If a user runs yaos-cli sync, the CLI should log "WARN: Running stateless sync. Downloading full vault history. For continuous updates, use 'yaos-cli daemon' to prevent rate-limiting."
• We should implement a bailout mechanism. When the CLI finishes syncing, it writes a tiny .yaos-state.json file to the local directory containing a single string: the server's current CRDT State Vector Hash or Generation ID. The next time the cronjob runs, the CLI sends that hash to the server.

4. We will rely on standard Node fs. I do not want to go down the rabbit hole of compiling and shipping C++ N-API binaries just to preserve birthtime like the official sync does. Chokidar and node:fs are good enough to be trusted for a Linux server.

To be clear, your PR is great as an alpha. My response should clear out the direction I want to take, and I'll mention any bugs in the PR itself.

In short,
1 - Yes
2 - Main repo
3 - Should wait
4 - I'll write below

[enieuwy]

Thanks for the clear direction — I agree on all four points.

I'll start on atomic writes (write-to-temp → fsync → rename) and the state-file bailout mechanism on the existing PR so that feedback is addressed. The PR stays in alpha shape until the VaultFS extraction lands.

Regarding #2 — I'd be happy to take a first pass at the client-side VaultFS interface in a week or so if that's helpful. I have a concrete picture of what the Node adapter needs from having built the CLI. Let me know if you'd prefer to drive it yourself or if you'd want me to propose something.

[kavinsood]

I'd be happy to take a first pass at the client-side VaultFS interface

That would be really helpful, feel free to take a pass at it! Just heads-up: because of the massive influx of users, my first priority is stabilizing the core engine and patching high-priority edge cases on main. I want to make sure the core is rock-solid before we execute a massive interface refactor.

Go ahead and draft that the extraction whenever you're ready. Once I'm done, I'll jump in too.

Now, about "Are there architectural invariants I'm missing that would make this harder than "replace VFS calls with fs calls"?"

I want to share a few edge cases I reasoned through that are not a concern, but you should know:

• An always-on headless daemon keeping a persistent WebSocket open could run up the cloud bill, but since the server uses Cloudflare's WebSocket Hibernation API (hibernate: true), the DO will safely sleep between the daemon's messages.
• The headless client uses the same out-of-band edit path, which uses fast-diff (applyDiffToYText), this generates almost character-level ops, so cursor preservation remains intact.
• Batch scripts like git checkout can technically spam the DO with simultaneous file updates, but the dirty-set dream loop in NodeDiskMirror handles this by coalescing event storms into backpressure-bounded batches. Make sure the same gates are there in the cli client.
• TOCTOU, atomic saves, and rename inference are implementation concerns for a good NodeDiskMirror. The draft PR already handles them (rename pair inference via content matching, chokidar awaitWriteFinish for stability, per-path write locks for serialization). They need to be solid, but they don't change the direction.
• The 500ms suppression window on slow filesystems, the content fingerprint check makes this sound on local filesystems. We're constraining to Linux + local fs. NFS/SMB is explicitly unsupported.

The invariants -

1. Your extracted normalizeVaultPath.ts handles slashes, but misses Unicode Normalization. macOS uses NFD decomposition (é = e + combining accent), while Linux uses NFC (single codepoint). Obsidian silently normalizes to NFC.
   If the CLI on Linux scans an rsync'd vault from a Mac, it will read NFD filenames, get a cache miss in the CRDT's pathToId map, and silently create a duplicate, orphaned Y.Text instance. The CLI's path normalizer must enforce .normalize('NFC') and strip trailing slashes, or cross-OS sync will corrupt.

2. The plugin gets O(delta) reconnects because IndexedDB holds the Y.Doc state. The draft PR uses a no-op persistence adapter. For a long-running daemon, this is fine. But if a user runs yaos-cli sync in a cronjob, it will force the DO to serialize and download the entire 50MB vault history every single minute, burning through Cloudflare compute limits instantly.
   Fix: After a sync completes, we must persist the encoded state via Y.encodeStateAsUpdate(doc) to a .yaos-state.bin file. On startup, the CLI loads this file, allowing the Yjs protocol to request only the O(delta) changes.

3. If two daemons (or a daemon and Obsidian) run on the same shared drive, they will both call ensureFile, generating colliding fileIds. The loser becomes a permanent orphan in the DO's memory. This must be heavily documented as a strict constraint.

4. If the server returns update_required, the plugin shows a UI banner. The CLI throws a fatal error. If managed by systemd with Restart=always, it will loop infinitely. The daemon needs to exit with a distinct code (e.g., process.exit(2)) so users can configure RestartPreventExitStatus=2.

Again, I appreciate the help!