From 3b5f8c8a2093677bf20952d811ec997a4650759b Mon Sep 17 00:00:00 2001 From: Razvan Laurus Date: Mon, 27 Apr 2026 10:29:46 +0200 Subject: [PATCH 01/11] docs(adr): add ADR-015 and migrate scala-sbt off virtiofs caches Closes #18. PR #20 raised virtiofsd's NOFILE ceiling 8x but virtiofsd --cache=auto accumulates backing-file FDs monotonically; any session long enough hits the new ceiling. The durable fix is to keep churning caches off virtiofs entirely. ADR-015 codifies the rule: plugins must not virtiofs-mount package-manager caches, build caches, or other LRU-shaped directories. Cache state lives on root.img (which persists per workspace), at the tool's default location or via env var redirect from a setup script. scala-sbt migrated as the reference: dropping the ~/.cache/coursier and ~/.ivy2 mounts is sufficient since both defaults are under \$HOME on root.img. Trade-off: first sbt update per workspace re-fetches; cache then stays warm for the workspace lifetime. Future work (called out in the ADR): VM snapshotting via qcow2 backing or cloud-hypervisor save/restore would obviate the warmup cost. Separately scoped. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/decisions/015-no-virtiofs-hot-caches.md | 52 ++++++++++++++++++++ plugins/scala-sbt/README.md | 11 ++++- plugins/scala-sbt/default.nix | 11 ----- 3 files changed, 61 insertions(+), 13 deletions(-) create mode 100644 docs/decisions/015-no-virtiofs-hot-caches.md diff --git a/docs/decisions/015-no-virtiofs-hot-caches.md b/docs/decisions/015-no-virtiofs-hot-caches.md new file mode 100644 index 0000000..ae4b9b0 --- /dev/null +++ b/docs/decisions/015-no-virtiofs-hot-caches.md @@ -0,0 +1,52 @@ +# 015: Don't virtiofs-mount churning cache directories + +**Date:** 2026-04-27 +**Status:** accepted + +## Problem + +Plugins have been mounting host package-manager caches (`~/.cache/coursier`, `~/.ivy2`) via virtiofs to share warm state with the host. Under FD-heavy workloads — `sbt update` validating hundreds of `.jar__sha1` entries in parallel, or any LRU-shaped cache traversal — `virtiofsd --cache=auto` accumulates backing-file FDs monotonically across the session and pins shares at the process `RLIMIT_NOFILE` ceiling. Host `EMFILE` surfaces in the guest as `ENFILE` ("Too many open files in system"), failing builds even when the guest has FDs to spare (#18). + +Raising the limit (PR #20, 65536 → 524288) buys headroom but doesn't change the monotonic accumulation. A long enough session hits the new ceiling. + +## Decision + +Plugins MUST NOT virtiofs-mount directories that churn under LRU-shaped access — package-manager caches, build-artifact caches, indexer state, log directories. Cache state lives on the guest's `root.img` (which persists per workspace across `up`/`down`) at the tool's default location; if the tool's default is not under `$HOME`, redirect via the relevant env var from a setup script. + +Reference table for common toolchains: + +| Tool | Default location | Env var | +| ------------------------ | ------------------------- | ---------------------------- | +| sbt / coursier | `~/.cache/coursier` | `COURSIER_CACHE` | +| ivy2 | `~/.ivy2/` | (default works) | +| npm / pnpm / yarn | `~/.npm`, `~/.pnpm-store` | `npm_config_cache`, `…` | +| pip / uv | `~/.cache/pip` | `PIP_CACHE_DIR` | +| cargo | `~/.cargo` | `CARGO_HOME` | +| go modules | `~/go/pkg/mod` | `GOMODCACHE` | +| docker storage | `/var/lib/docker` | (already guest-native) | + +For tools whose default cache lives under `$HOME`, the simplest implementation is to *not mount* — the default already lands on `root.img`. + +Virtiofs continues to be the right choice for source trees, credentials (`~/.env`), small config (`.ssh`, `.claude`), and any cross-boundary share with moderate file count and genuine cross-boundary semantics. + +## Rationale + +Virtiofs has two costs that scale with workload, not share size: + +1. **`virtiofsd` holds backing-file FDs for every file it serves under `--cache=auto`.** The high-water mark grows monotonically across a long session; LRU caches with thousands of small files (`.jar__sha1`) hit the ceiling fastest. +2. **Cross-boundary semantics for atomic operations are constrained vs. native fs** (see ADR-001 on `O_TMPFILE`). Package managers rely on rename atomicity and tmpfile semantics for partial-write cleanup. + +A churning cache fails both tests: many small files, and the tool relies on filesystem invariants. Putting it on virtiofs provides no semantic value (the cache is a derived artifact, not a source of truth) and pays the ongoing FD cost. + +The host-cache-sharing argument — "I want my IDE outside the VM to use the same jars" — is real but trades a one-time re-download per workspace against a soft architectural ceiling. The trade-off favors guest-native: `root.img` keeps the cache warm within a workspace, and the user's bandwidth amortizes a fresh download once. + +## Consequences + +- New plugins do not declare `mounts` for cache paths. Tools either pick up defaults under `$HOME` or get redirected via env vars from `scripts/setup.sh` (since plugins cannot inject `env`, per ADR-013). +- Users lose live host↔guest cache sharing. First fetch per workspace re-downloads. +- `plugins/scala-sbt` migrated in this PR: drops `~/.cache/coursier` and `~/.ivy2` mounts. Coursier's default `~/.cache/coursier` and ivy2's default `~/.ivy2/` land on `root.img` automatically. +- A future `nixbox doctor` lint can flag plugins mounting common cache paths. + +## Future work + +VM snapshotting (cold disk clone via qcow2 backing file, or live save/restore via cloud-hypervisor's HTTP API) would make the warmup cost disappear: a "blessed" warm image starts new VMs in seconds with caches pre-populated. This is a separate, larger feature; the decision above is the immediate fix. diff --git a/plugins/scala-sbt/README.md b/plugins/scala-sbt/README.md index 030b631..a41d836 100644 --- a/plugins/scala-sbt/README.md +++ b/plugins/scala-sbt/README.md @@ -12,10 +12,17 @@ Scala toolchain with optional private Maven/Nexus credentials. | Category | Details | |---|---| -| **Packages** | `sbt`, `scala` | -| **Mounts** | `~/.cache/coursier`, `~/.ivy2` (only if they exist on host) | +| **Packages** | `sbt`, `scala`, `scala-cli` | | **Domains** | `maven.org`, `scala-sbt.org`, plus `MAVEN_REPO_HOST` if set | +## Caches + +Coursier and ivy2 caches live on the guest's `root.img` at default paths +(`~/.cache/coursier`, `~/.ivy2`). They persist across `up`/`down` for the same +workspace but are **not** shared with the host (see +[ADR-015](../../docs/decisions/015-no-virtiofs-hot-caches.md)). First +`sbt update` per workspace re-downloads dependencies. + ## Private repository credentials Set all three env vars to auto-configure `~/.sbt/1.0/credentials.sbt`: diff --git a/plugins/scala-sbt/default.nix b/plugins/scala-sbt/default.nix index ad92d47..b001c9f 100644 --- a/plugins/scala-sbt/default.nix +++ b/plugins/scala-sbt/default.nix @@ -1,10 +1,3 @@ -let - home = builtins.getEnv "HOME"; - mountIf = path: target: - if home != "" && builtins.pathExists (home + path) - then [ { source = "~" + path; inherit target; } ] - else [ ]; -in { nix.packages = [ "sbt" @@ -12,10 +5,6 @@ in "scala-cli" ]; - mounts = - mountIf "/.cache/coursier" "~/.cache/coursier" - ++ mountIf "/.ivy2" "~/.ivy2"; - network.domains = [ "maven.org" From 9d1d6bcc2b18c79ed64bfff96b4b4af70f687e64 Mon Sep 17 00:00:00 2001 From: Razvan Laurus Date: Mon, 27 Apr 2026 10:34:44 +0200 Subject: [PATCH 02/11] docs(adr-015): correct credential channel reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Credentials are not virtiofs-mounted — they're written to a one-shot ext4 disk that the guest reads at boot and unmounts (ADR-003). Listing ~/.env as a virtiofs example contradicts the architecture and the explicit-credentials philosophy (ADR-013). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/decisions/015-no-virtiofs-hot-caches.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/decisions/015-no-virtiofs-hot-caches.md b/docs/decisions/015-no-virtiofs-hot-caches.md index ae4b9b0..eb9b0ec 100644 --- a/docs/decisions/015-no-virtiofs-hot-caches.md +++ b/docs/decisions/015-no-virtiofs-hot-caches.md @@ -27,7 +27,7 @@ Reference table for common toolchains: For tools whose default cache lives under `$HOME`, the simplest implementation is to *not mount* — the default already lands on `root.img`. -Virtiofs continues to be the right choice for source trees, credentials (`~/.env`), small config (`.ssh`, `.claude`), and any cross-boundary share with moderate file count and genuine cross-boundary semantics. +Virtiofs continues to be the right choice for source trees and small config directories the user explicitly mounts (`~/.ssh`, `~/.claude`, etc.) — moderate file count with genuine cross-boundary semantics. Credentials use a separate one-shot ext4 disk read at boot (ADR-003), not virtiofs. ## Rationale From 11244778a723d78ee77bdebbd54b74d56a17c681 Mon Sep 17 00:00:00 2001 From: Razvan Laurus Date: Mon, 27 Apr 2026 10:36:19 +0200 Subject: [PATCH 03/11] docs(adr-015): drop ~/.ssh example; clarify sensitive-data channels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ~/.ssh is not a recommended virtiofs target — the architecture explicitly avoids sharing host SSH identity. ADR-014 injects a per-VM key for VM SSH identity, and host SSH keys (GitHub, AWS, deploy keys) deliberately do not cross the sandbox boundary. Reframe the appropriate-virtiofs-uses paragraph to state the principle (moderate file count, in-place cross-boundary semantics) and point at dedicated channels for sensitive data: ADR-003 (credentials disk), ADR-014 (SSH key injection). Plugins mounting additional host state (claude-code/~/.claude) own that tradeoff in their own ADR (ADR-010). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/decisions/015-no-virtiofs-hot-caches.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/decisions/015-no-virtiofs-hot-caches.md b/docs/decisions/015-no-virtiofs-hot-caches.md index eb9b0ec..605d69e 100644 --- a/docs/decisions/015-no-virtiofs-hot-caches.md +++ b/docs/decisions/015-no-virtiofs-hot-caches.md @@ -27,7 +27,7 @@ Reference table for common toolchains: For tools whose default cache lives under `$HOME`, the simplest implementation is to *not mount* — the default already lands on `root.img`. -Virtiofs continues to be the right choice for source trees and small config directories the user explicitly mounts (`~/.ssh`, `~/.claude`, etc.) — moderate file count with genuine cross-boundary semantics. Credentials use a separate one-shot ext4 disk read at boot (ADR-003), not virtiofs. +Virtiofs continues to be the right choice for source trees and similar shares with moderate file count where in-place semantics matter (host editor and guest tool reading the same files). Sensitive data uses dedicated explicit channels: credentials via the one-shot ext4 disk (ADR-003), VM SSH identity via per-VM key injection (ADR-014). Plugins that mount additional host state (e.g. claude-code's `~/.claude`, ADR-010) own the tradeoff in their own ADR. ## Rationale From 0e99054c81af37fd6a788476174121034cbd166a Mon Sep 17 00:00:00 2001 From: Razvan Laurus Date: Mon, 27 Apr 2026 10:45:01 +0200 Subject: [PATCH 04/11] feat(scala-sbt): warm host caches via bootstrap mount + rsync ADR-015 distinguishes steady-state runtime virtiofs use (forbidden for churning caches) from bounded one-shot bootstrap reads (permitted). scala-sbt adopts the bootstrap pattern: - RO virtiofs mounts at /mnt/host-cache/coursier and /mnt/host-cache/ivy2 (only if host paths exist) - Setup script rsyncs into ~/.cache/coursier and ~/.ivy2 on first boot, guarded by a sentinel file - Subsequent boots: rsync is a no-op; sbt's runtime I/O lives entirely on root.img and never crosses virtiofs - rsync added to nix.packages Trade-off: virtiofsd holds backing-file FDs accumulated during the rsync pass for the VM's lifetime (~50-100k for typical coursier caches), well under the 524288 ceiling from PR #20. No further accumulation occurs since nothing reads the mount after warmup. ADR-015 updated to codify the bootstrap exception with explicit MUST requirements (readonly, side path, sentinel-guarded, no post-warmup access). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/decisions/015-no-virtiofs-hot-caches.md | 35 +++++++++++++------- plugins/scala-sbt/README.md | 16 ++++++--- plugins/scala-sbt/default.nix | 15 +++++++++ plugins/scala-sbt/scripts/setup.sh | 15 +++++++++ 4 files changed, 64 insertions(+), 17 deletions(-) diff --git a/docs/decisions/015-no-virtiofs-hot-caches.md b/docs/decisions/015-no-virtiofs-hot-caches.md index 605d69e..a961d7c 100644 --- a/docs/decisions/015-no-virtiofs-hot-caches.md +++ b/docs/decisions/015-no-virtiofs-hot-caches.md @@ -1,17 +1,17 @@ -# 015: Don't virtiofs-mount churning cache directories +# 015: No steady-state virtiofs for churning caches **Date:** 2026-04-27 **Status:** accepted ## Problem -Plugins have been mounting host package-manager caches (`~/.cache/coursier`, `~/.ivy2`) via virtiofs to share warm state with the host. Under FD-heavy workloads — `sbt update` validating hundreds of `.jar__sha1` entries in parallel, or any LRU-shaped cache traversal — `virtiofsd --cache=auto` accumulates backing-file FDs monotonically across the session and pins shares at the process `RLIMIT_NOFILE` ceiling. Host `EMFILE` surfaces in the guest as `ENFILE` ("Too many open files in system"), failing builds even when the guest has FDs to spare (#18). +Plugins have been mounting host package-manager caches (`~/.cache/coursier`, `~/.ivy2`) via virtiofs as the runtime read/write path. Under FD-heavy workloads — `sbt update` validating hundreds of `.jar__sha1` entries in parallel, or any LRU-shaped cache traversal — `virtiofsd --cache=auto` accumulates backing-file FDs monotonically across the session and pins shares at the process `RLIMIT_NOFILE` ceiling. Host `EMFILE` surfaces in the guest as `ENFILE` ("Too many open files in system"), failing builds even when the guest has FDs to spare (#18). Raising the limit (PR #20, 65536 → 524288) buys headroom but doesn't change the monotonic accumulation. A long enough session hits the new ceiling. ## Decision -Plugins MUST NOT virtiofs-mount directories that churn under LRU-shaped access — package-manager caches, build-artifact caches, indexer state, log directories. Cache state lives on the guest's `root.img` (which persists per workspace across `up`/`down`) at the tool's default location; if the tool's default is not under `$HOME`, redirect via the relevant env var from a setup script. +Plugins MUST NOT use virtiofs as the **steady-state runtime path** for churning data — package-manager caches, build-artifact caches, indexer state, log directories. Runtime cache I/O lives on the guest's `root.img` (which persists per workspace across `up`/`down`) at the tool's default location. If the default is not under `$HOME`, redirect via the relevant env var from a setup script. Reference table for common toolchains: @@ -25,7 +25,17 @@ Reference table for common toolchains: | go modules | `~/go/pkg/mod` | `GOMODCACHE` | | docker storage | `/var/lib/docker` | (already guest-native) | -For tools whose default cache lives under `$HOME`, the simplest implementation is to *not mount* — the default already lands on `root.img`. +### Bootstrap exception + +A read-only virtiofs mount at a side path used for a one-shot copy into `root.img` during the setup script is permitted. Once warmup completes, the mount stops being accessed and the runtime tool I/O lives entirely on `root.img`. The `virtiofsd` FD high-water mark is bounded by the number of files copied (one-time peak), not by session length — typical coursier caches yield ~50–100k FDs, comfortably under the 524288 ceiling. + +Bootstrap mounts MUST: +- Be marked `readonly = true`. +- Mount at a side path (e.g. `/mnt/host-cache/coursier`), never at the tool's runtime cache location. +- Be read by an idempotent setup script that writes a sentinel and skips re-runs. +- Not be touched after the warmup completes (no live host↔guest cache sharing). + +### What stays on virtiofs Virtiofs continues to be the right choice for source trees and similar shares with moderate file count where in-place semantics matter (host editor and guest tool reading the same files). Sensitive data uses dedicated explicit channels: credentials via the one-shot ext4 disk (ADR-003), VM SSH identity via per-VM key injection (ADR-014). Plugins that mount additional host state (e.g. claude-code's `~/.claude`, ADR-010) own the tradeoff in their own ADR. @@ -33,20 +43,21 @@ Virtiofs continues to be the right choice for source trees and similar shares wi Virtiofs has two costs that scale with workload, not share size: -1. **`virtiofsd` holds backing-file FDs for every file it serves under `--cache=auto`.** The high-water mark grows monotonically across a long session; LRU caches with thousands of small files (`.jar__sha1`) hit the ceiling fastest. +1. **`virtiofsd` holds backing-file FDs for every file it serves under `--cache=auto`.** The high-water mark grows monotonically with use. 2. **Cross-boundary semantics for atomic operations are constrained vs. native fs** (see ADR-001 on `O_TMPFILE`). Package managers rely on rename atomicity and tmpfile semantics for partial-write cleanup. -A churning cache fails both tests: many small files, and the tool relies on filesystem invariants. Putting it on virtiofs provides no semantic value (the cache is a derived artifact, not a source of truth) and pays the ongoing FD cost. +A steady-state churning cache fails both tests: many small files, and the tool relies on filesystem invariants. The bootstrap pattern accepts the first cost briefly (one rsync pass, bounded by file count) to retain a useful property — first-build is fast because the host's warm cache transfers in. Steady-state cache I/O then runs against `root.img` with no virtiofs involvement. -The host-cache-sharing argument — "I want my IDE outside the VM to use the same jars" — is real but trades a one-time re-download per workspace against a soft architectural ceiling. The trade-off favors guest-native: `root.img` keeps the cache warm within a workspace, and the user's bandwidth amortizes a fresh download once. +The host-cache-sharing argument — "I want my IDE outside the VM to use the same jars" — is partially preserved by bootstrap: the guest gets the host's current cache state at workspace creation, then diverges. Live cross-boundary sharing is given up; one-time seeding plus per-workspace persistence is enough in practice. ## Consequences -- New plugins do not declare `mounts` for cache paths. Tools either pick up defaults under `$HOME` or get redirected via env vars from `scripts/setup.sh` (since plugins cannot inject `env`, per ADR-013). -- Users lose live host↔guest cache sharing. First fetch per workspace re-downloads. -- `plugins/scala-sbt` migrated in this PR: drops `~/.cache/coursier` and `~/.ivy2` mounts. Coursier's default `~/.cache/coursier` and ivy2's default `~/.ivy2/` land on `root.img` automatically. -- A future `nixbox doctor` lint can flag plugins mounting common cache paths. +- New plugins do not declare runtime mounts for cache paths. Defaults under `$HOME` work as-is; non-`$HOME` defaults get an env-var redirect from `scripts/setup.sh` (since plugins cannot inject `env`, per ADR-013). +- Plugins MAY declare RO bootstrap mounts at side paths and rsync from them in the setup script. +- Users lose live host↔guest cache sharing. First boot per workspace warms the cache from host; subsequent boots use `root.img` directly. +- `plugins/scala-sbt` adopts the bootstrap pattern in this PR: RO mounts at `/mnt/host-cache/coursier` and `/mnt/host-cache/ivy2`, rsynced into `~/.cache/coursier` and `~/.ivy2` on first boot, sentinel-guarded for idempotency. The plugin adds `rsync` to its packages. +- A future `nixbox doctor` lint can flag plugins that mount cache paths at runtime locations (rather than side paths). ## Future work -VM snapshotting (cold disk clone via qcow2 backing file, or live save/restore via cloud-hypervisor's HTTP API) would make the warmup cost disappear: a "blessed" warm image starts new VMs in seconds with caches pre-populated. This is a separate, larger feature; the decision above is the immediate fix. +VM snapshotting (cold disk clone via qcow2 backing file, or live save/restore via cloud-hypervisor's HTTP API) would make even the bootstrap cost disappear: a "blessed" warm image starts new VMs in seconds with caches pre-populated, and replaces the rsync entirely. This is a separate, larger feature; the decision above is the immediate fix. diff --git a/plugins/scala-sbt/README.md b/plugins/scala-sbt/README.md index a41d836..0bc3b86 100644 --- a/plugins/scala-sbt/README.md +++ b/plugins/scala-sbt/README.md @@ -12,16 +12,22 @@ Scala toolchain with optional private Maven/Nexus credentials. | Category | Details | |---|---| -| **Packages** | `sbt`, `scala`, `scala-cli` | +| **Packages** | `sbt`, `scala`, `scala-cli`, `rsync` | | **Domains** | `maven.org`, `scala-sbt.org`, plus `MAVEN_REPO_HOST` if set | ## Caches Coursier and ivy2 caches live on the guest's `root.img` at default paths -(`~/.cache/coursier`, `~/.ivy2`). They persist across `up`/`down` for the same -workspace but are **not** shared with the host (see -[ADR-015](../../docs/decisions/015-no-virtiofs-hot-caches.md)). First -`sbt update` per workspace re-downloads dependencies. +(`~/.cache/coursier`, `~/.ivy2`). On first boot per workspace, the plugin +warms them by `rsync`-copying from the host's `~/.cache/coursier` and +`~/.ivy2` (mounted read-only at `/mnt/host-cache/...` for the boot, then +idle). Subsequent boots skip the warmup; sbt's runtime I/O lives entirely +on `root.img` and never crosses virtiofs. + +If the host paths don't exist, the warmup is skipped and the cache is +populated by `sbt update` over the network. See +[ADR-015](../../docs/decisions/015-no-virtiofs-hot-caches.md) for the +rationale. ## Private repository credentials diff --git a/plugins/scala-sbt/default.nix b/plugins/scala-sbt/default.nix index b001c9f..fdda74b 100644 --- a/plugins/scala-sbt/default.nix +++ b/plugins/scala-sbt/default.nix @@ -1,10 +1,25 @@ +let + home = builtins.getEnv "HOME"; + # Bootstrap mount: RO virtiofs share read once at boot by setup.sh, then + # idle for the rest of the VM's lifetime (see ADR-015). Skipped if the + # host path doesn't exist. + warmFromHost = src: target: + if home != "" && builtins.pathExists (home + src) + then [ { source = "~" + src; inherit target; readonly = true; } ] + else [ ]; +in { nix.packages = [ "sbt" "scala" "scala-cli" + "rsync" ]; + mounts = + warmFromHost "/.cache/coursier" "/mnt/host-cache/coursier" + ++ warmFromHost "/.ivy2" "/mnt/host-cache/ivy2"; + network.domains = [ "maven.org" diff --git a/plugins/scala-sbt/scripts/setup.sh b/plugins/scala-sbt/scripts/setup.sh index 398bf8c..90a76ba 100755 --- a/plugins/scala-sbt/scripts/setup.sh +++ b/plugins/scala-sbt/scripts/setup.sh @@ -11,3 +11,18 @@ user=$MAVEN_REPO_USER password=$MAVEN_REPO_PASSWORD EOF fi + +# One-shot warmup of host caches into root.img on first boot per workspace. +# Runtime sbt I/O lives entirely on root.img (see ADR-015). +warm_cache() { + local src="$1" dst="$2" name="$3" + [ -d "$src" ] || return 0 + [ -f "$dst/.nixbox-warmed" ] && return 0 + echo "==> Warming $name cache from host (one-time per workspace)..." + mkdir -p "$dst" + rsync -a "$src/" "$dst/" + touch "$dst/.nixbox-warmed" +} + +warm_cache /mnt/host-cache/coursier "$HOME/.cache/coursier" coursier +warm_cache /mnt/host-cache/ivy2 "$HOME/.ivy2" ivy2 From c00789c2abfc7f97a95524f5b8eef12d2c20c1e3 Mon Sep 17 00:00:00 2001 From: Razvan Laurus Date: Mon, 27 Apr 2026 10:51:01 +0200 Subject: [PATCH 05/11] docs(adr-015): trim to essentials Remove cruft that doesn't drive the decision: - toolchain reference table (belongs in a plugin-authoring guide) - "what stays on virtiofs" sub-section (scope creep into ADR-003/014/010) - atomic-ops rationale (collapsed; ADR-001 owns it) - host-cache-sharing trade-off discussion (bootstrap addresses it) - "log directories" example (logs are append-mostly, not LRU) - future-work detail (one-line callout) Keep what matters for a plugin author reading this in a year: the steady-state prohibition, the bootstrap MUSTs, the FD rationale, and scala-sbt as concrete grounding. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/decisions/015-no-virtiofs-hot-caches.md | 48 +++++--------------- 1 file changed, 12 insertions(+), 36 deletions(-) diff --git a/docs/decisions/015-no-virtiofs-hot-caches.md b/docs/decisions/015-no-virtiofs-hot-caches.md index a961d7c..3f65918 100644 --- a/docs/decisions/015-no-virtiofs-hot-caches.md +++ b/docs/decisions/015-no-virtiofs-hot-caches.md @@ -5,59 +5,35 @@ ## Problem -Plugins have been mounting host package-manager caches (`~/.cache/coursier`, `~/.ivy2`) via virtiofs as the runtime read/write path. Under FD-heavy workloads — `sbt update` validating hundreds of `.jar__sha1` entries in parallel, or any LRU-shaped cache traversal — `virtiofsd --cache=auto` accumulates backing-file FDs monotonically across the session and pins shares at the process `RLIMIT_NOFILE` ceiling. Host `EMFILE` surfaces in the guest as `ENFILE` ("Too many open files in system"), failing builds even when the guest has FDs to spare (#18). - -Raising the limit (PR #20, 65536 → 524288) buys headroom but doesn't change the monotonic accumulation. A long enough session hits the new ceiling. +`virtiofsd --cache=auto` accumulates backing-file FDs monotonically; under FD-heavy workloads (e.g. `sbt update` validating thousands of `.jar__sha1` files) host shares pin at the process `RLIMIT_NOFILE` ceiling and surface as `ENFILE` in the guest (#18). Raising the limit (PR #20, 524288) buys headroom but doesn't change the monotonicity. ## Decision -Plugins MUST NOT use virtiofs as the **steady-state runtime path** for churning data — package-manager caches, build-artifact caches, indexer state, log directories. Runtime cache I/O lives on the guest's `root.img` (which persists per workspace across `up`/`down`) at the tool's default location. If the default is not under `$HOME`, redirect via the relevant env var from a setup script. - -Reference table for common toolchains: +Plugins MUST NOT use virtiofs as the **steady-state runtime path** for churning data — package-manager caches, build-artifact caches, indexer state. Runtime cache I/O lives on `root.img`, which persists per workspace, at the tool's default location (or via env-var redirect from a setup script when the default is not under `$HOME`). -| Tool | Default location | Env var | -| ------------------------ | ------------------------- | ---------------------------- | -| sbt / coursier | `~/.cache/coursier` | `COURSIER_CACHE` | -| ivy2 | `~/.ivy2/` | (default works) | -| npm / pnpm / yarn | `~/.npm`, `~/.pnpm-store` | `npm_config_cache`, `…` | -| pip / uv | `~/.cache/pip` | `PIP_CACHE_DIR` | -| cargo | `~/.cargo` | `CARGO_HOME` | -| go modules | `~/go/pkg/mod` | `GOMODCACHE` | -| docker storage | `/var/lib/docker` | (already guest-native) | +Use virtiofs only where in-place cross-boundary semantics matter (e.g. source trees). ### Bootstrap exception -A read-only virtiofs mount at a side path used for a one-shot copy into `root.img` during the setup script is permitted. Once warmup completes, the mount stops being accessed and the runtime tool I/O lives entirely on `root.img`. The `virtiofsd` FD high-water mark is bounded by the number of files copied (one-time peak), not by session length — typical coursier caches yield ~50–100k FDs, comfortably under the 524288 ceiling. +A read-only virtiofs mount used for a one-shot copy into `root.img` during the setup script is permitted. The FD high-water mark is bounded by the file count copied (one-time peak), not by session length. Bootstrap mounts MUST: -- Be marked `readonly = true`. +- Be `readonly = true`. - Mount at a side path (e.g. `/mnt/host-cache/coursier`), never at the tool's runtime cache location. - Be read by an idempotent setup script that writes a sentinel and skips re-runs. -- Not be touched after the warmup completes (no live host↔guest cache sharing). - -### What stays on virtiofs - -Virtiofs continues to be the right choice for source trees and similar shares with moderate file count where in-place semantics matter (host editor and guest tool reading the same files). Sensitive data uses dedicated explicit channels: credentials via the one-shot ext4 disk (ADR-003), VM SSH identity via per-VM key injection (ADR-014). Plugins that mount additional host state (e.g. claude-code's `~/.claude`, ADR-010) own the tradeoff in their own ADR. +- Not be touched after warmup (no live host↔guest cache sharing). ## Rationale -Virtiofs has two costs that scale with workload, not share size: - -1. **`virtiofsd` holds backing-file FDs for every file it serves under `--cache=auto`.** The high-water mark grows monotonically with use. -2. **Cross-boundary semantics for atomic operations are constrained vs. native fs** (see ADR-001 on `O_TMPFILE`). Package managers rely on rename atomicity and tmpfile semantics for partial-write cleanup. - -A steady-state churning cache fails both tests: many small files, and the tool relies on filesystem invariants. The bootstrap pattern accepts the first cost briefly (one rsync pass, bounded by file count) to retain a useful property — first-build is fast because the host's warm cache transfers in. Steady-state cache I/O then runs against `root.img` with no virtiofs involvement. - -The host-cache-sharing argument — "I want my IDE outside the VM to use the same jars" — is partially preserved by bootstrap: the guest gets the host's current cache state at workspace creation, then diverges. Live cross-boundary sharing is given up; one-time seeding plus per-workspace persistence is enough in practice. +`virtiofsd --cache=auto` holds a backing-file FD per served file and never meaningfully drops them — the FD set grows monotonically with workload. A persistent mount of a churning cache fails on this alone; the constrained atomic-op semantics noted in ADR-001 are an aggravating secondary cost. Bootstrap mounts pay the FD cost once (bounded by file count), then go quiet. ## Consequences -- New plugins do not declare runtime mounts for cache paths. Defaults under `$HOME` work as-is; non-`$HOME` defaults get an env-var redirect from `scripts/setup.sh` (since plugins cannot inject `env`, per ADR-013). -- Plugins MAY declare RO bootstrap mounts at side paths and rsync from them in the setup script. -- Users lose live host↔guest cache sharing. First boot per workspace warms the cache from host; subsequent boots use `root.img` directly. -- `plugins/scala-sbt` adopts the bootstrap pattern in this PR: RO mounts at `/mnt/host-cache/coursier` and `/mnt/host-cache/ivy2`, rsynced into `~/.cache/coursier` and `~/.ivy2` on first boot, sentinel-guarded for idempotency. The plugin adds `rsync` to its packages. -- A future `nixbox doctor` lint can flag plugins that mount cache paths at runtime locations (rather than side paths). +- New plugins do not declare runtime mounts for cache paths. Defaults under `$HOME` work; non-`$HOME` defaults get an env-var redirect from `scripts/setup.sh` (plugins cannot inject `env`, per ADR-013). +- Plugins MAY declare RO bootstrap mounts at side paths and rsync from them in setup. +- Live host↔guest cache sharing is gone; first boot per workspace warms from host, subsequent boots use `root.img` directly. +- `plugins/scala-sbt` adopts the bootstrap pattern: RO mounts at `/mnt/host-cache/{coursier,ivy2}`, rsynced into `~/.cache/coursier` and `~/.ivy2` on first boot, sentinel-guarded. Adds `rsync` to its packages. ## Future work -VM snapshotting (cold disk clone via qcow2 backing file, or live save/restore via cloud-hypervisor's HTTP API) would make even the bootstrap cost disappear: a "blessed" warm image starts new VMs in seconds with caches pre-populated, and replaces the rsync entirely. This is a separate, larger feature; the decision above is the immediate fix. +VM snapshotting would replace the bootstrap rsync with a pre-warmed image; tracked separately. From 7313b0849b1c49d1d70f051ec2fae3724d05a90d Mon Sep 17 00:00:00 2001 From: Razvan Laurus Date: Mon, 27 Apr 2026 10:57:42 +0200 Subject: [PATCH 06/11] feat(scala-sbt): release bootstrap mounts after warmup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bootstrap mounts (RO virtiofs at /mnt/host-cache/*) were left attached for the VM's lifetime, leaving virtiofsd processes idle but holding the backing-file FDs accumulated during the rsync. ADR-015's whole pitch is "no steady-state virtiofs"; leaving virtiofsd idling walks back from that. Changes: - bin/nixbox: do_unmount is idempotent — if no active mount matches the target, log and return 0 instead of dying. Lets post-up hooks run unconditionally on every up (warmup is sentinel-guarded; mount may not have been (re)created on subsequent boots). - plugins/scala-sbt: post-up hook releases /mnt/host-cache/{coursier,ivy2} via nixbox unmount. nixbox unmount detaches the virtio device and kills virtiofsd, freeing the accumulated FDs. - ADR-015: split the bootstrap rules into MUSTs (readonly, side path, sentinel, no post-warmup access) and SHOULDs (release after warmup). In-tree plugins MUST release; third-party plugins are encouraged but not required, since a single bootstrap's FDs are well under the 524288 ceiling. Co-Authored-By: Claude Opus 4.7 (1M context) --- bin/nixbox | 2 +- docs/decisions/015-no-virtiofs-hot-caches.md | 5 ++++- plugins/scala-sbt/README.md | 6 +++--- plugins/scala-sbt/default.nix | 7 +++++++ 4 files changed, 15 insertions(+), 5 deletions(-) diff --git a/bin/nixbox b/bin/nixbox index 6830a7b..70215be 100755 --- a/bin/nixbox +++ b/bin/nixbox @@ -595,7 +595,7 @@ do_unmount() { fi done - [ -z "$mount_idx" ] && die "No mount found for target '$target' on VM '$name'" + [ -z "$mount_idx" ] && { log_sub "No active mount at '$target' on VM '$name' (already released?)"; return 0; } ssh "${SSH_OPTS[@]}" ${VM_USER}@"$TAP_GUEST_IP" "sudo umount '$target'" diff --git a/docs/decisions/015-no-virtiofs-hot-caches.md b/docs/decisions/015-no-virtiofs-hot-caches.md index 3f65918..6729c06 100644 --- a/docs/decisions/015-no-virtiofs-hot-caches.md +++ b/docs/decisions/015-no-virtiofs-hot-caches.md @@ -23,6 +23,8 @@ Bootstrap mounts MUST: - Be read by an idempotent setup script that writes a sentinel and skips re-runs. - Not be touched after warmup (no live host↔guest cache sharing). +Bootstrap mounts SHOULD be released after warmup via a `post-up` hook (`nixbox unmount ` is idempotent and frees the virtiofsd FDs accumulated during the copy). **In-tree plugins MUST release.** Third-party plugins are encouraged to release but are not required to — the FD high-water from a single bootstrap is well under the 524288 ceiling. + ## Rationale `virtiofsd --cache=auto` holds a backing-file FD per served file and never meaningfully drops them — the FD set grows monotonically with workload. A persistent mount of a churning cache fails on this alone; the constrained atomic-op semantics noted in ADR-001 are an aggravating secondary cost. Bootstrap mounts pay the FD cost once (bounded by file count), then go quiet. @@ -32,7 +34,8 @@ Bootstrap mounts MUST: - New plugins do not declare runtime mounts for cache paths. Defaults under `$HOME` work; non-`$HOME` defaults get an env-var redirect from `scripts/setup.sh` (plugins cannot inject `env`, per ADR-013). - Plugins MAY declare RO bootstrap mounts at side paths and rsync from them in setup. - Live host↔guest cache sharing is gone; first boot per workspace warms from host, subsequent boots use `root.img` directly. -- `plugins/scala-sbt` adopts the bootstrap pattern: RO mounts at `/mnt/host-cache/{coursier,ivy2}`, rsynced into `~/.cache/coursier` and `~/.ivy2` on first boot, sentinel-guarded. Adds `rsync` to its packages. +- `plugins/scala-sbt` adopts the bootstrap pattern: RO mounts at `/mnt/host-cache/{coursier,ivy2}`, rsynced into `~/.cache/coursier` and `~/.ivy2` on first boot, sentinel-guarded; mounts released via `post-up` hook. Adds `rsync` to its packages. +- `nixbox unmount` is idempotent (no-op + log if the target is not mounted), so post-up hooks are safe to run unconditionally on every `up`. ## Future work diff --git a/plugins/scala-sbt/README.md b/plugins/scala-sbt/README.md index 0bc3b86..ad33bbf 100644 --- a/plugins/scala-sbt/README.md +++ b/plugins/scala-sbt/README.md @@ -20,9 +20,9 @@ Scala toolchain with optional private Maven/Nexus credentials. Coursier and ivy2 caches live on the guest's `root.img` at default paths (`~/.cache/coursier`, `~/.ivy2`). On first boot per workspace, the plugin warms them by `rsync`-copying from the host's `~/.cache/coursier` and -`~/.ivy2` (mounted read-only at `/mnt/host-cache/...` for the boot, then -idle). Subsequent boots skip the warmup; sbt's runtime I/O lives entirely -on `root.img` and never crosses virtiofs. +`~/.ivy2` (mounted read-only at `/mnt/host-cache/...`). The mounts are +released via `post-up` hook once warmup completes — `virtiofsd` exits and +the runtime sbt I/O lives entirely on `root.img`. If the host paths don't exist, the warmup is skipped and the cache is populated by `sbt update` over the network. See diff --git a/plugins/scala-sbt/default.nix b/plugins/scala-sbt/default.nix index fdda74b..b3bc01a 100644 --- a/plugins/scala-sbt/default.nix +++ b/plugins/scala-sbt/default.nix @@ -31,4 +31,11 @@ in ); scripts = [ ./scripts/setup.sh ]; + + # Release bootstrap mounts after warmup (ADR-015): drops the guest mount, + # detaches the virtio device, kills virtiofsd → frees its accumulated FDs. + hooks.post-up = [ + "nixbox unmount /mnt/host-cache/coursier" + "nixbox unmount /mnt/host-cache/ivy2" + ]; } From 5213865304578588251596fd8b0736c0df2c7f06 Mon Sep 17 00:00:00 2001 From: Razvan Laurus Date: Mon, 27 Apr 2026 11:05:06 +0200 Subject: [PATCH 07/11] refactor(scala-sbt): warm cache via plugin command, drop virtiofs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the virtiofs-bootstrap pattern with a host-side plugin command following the established 'nixbox aws login' / 'nixbox claude-code sync-config' pattern. - New: plugins/scala-sbt/commands/warm-cache.sh — pipes host's ~/.cache/coursier and ~/.ivy2 into the guest via 'tar | nixbox run "tar -x"' over the existing SSH channel. Sentinel-guarded for idempotency. - Plugin: post-up hook calls 'nixbox scala-sbt warm-cache'. No virtiofs mounts at all. No rsync in nix.packages. Setup script reverts to credentials-only. - ADR-015: drops the bootstrap-exception entirely. The decision is now strictly "no virtiofs for churning caches"; warmup is delivered by a host-side plugin command, not a transient mount. - bin/nixbox: revert the do_unmount idempotency change — it was scaffolding for the previous post-up unmount approach, no longer needed. Trade-off vs. the previous virtiofs-bootstrap approach: tar over SSH is slightly slower than virtiofs+rsync (~1.5x for local vsock) but eliminates the virtiofsd FD legacy entirely — no idle processes, no held FDs, no exception in the ADR. The plugin command is also user-invokable for manual refreshes. Co-Authored-By: Claude Opus 4.7 (1M context) --- bin/nixbox | 2 +- docs/decisions/015-no-virtiofs-hot-caches.md | 31 +++++++------------- plugins/scala-sbt/README.md | 21 +++++++------ plugins/scala-sbt/commands/warm-cache.sh | 31 ++++++++++++++++++++ plugins/scala-sbt/default.nix | 24 ++------------- plugins/scala-sbt/scripts/setup.sh | 15 ---------- 6 files changed, 58 insertions(+), 66 deletions(-) create mode 100755 plugins/scala-sbt/commands/warm-cache.sh diff --git a/bin/nixbox b/bin/nixbox index 70215be..6830a7b 100755 --- a/bin/nixbox +++ b/bin/nixbox @@ -595,7 +595,7 @@ do_unmount() { fi done - [ -z "$mount_idx" ] && { log_sub "No active mount at '$target' on VM '$name' (already released?)"; return 0; } + [ -z "$mount_idx" ] && die "No mount found for target '$target' on VM '$name'" ssh "${SSH_OPTS[@]}" ${VM_USER}@"$TAP_GUEST_IP" "sudo umount '$target'" diff --git a/docs/decisions/015-no-virtiofs-hot-caches.md b/docs/decisions/015-no-virtiofs-hot-caches.md index 6729c06..04944c0 100644 --- a/docs/decisions/015-no-virtiofs-hot-caches.md +++ b/docs/decisions/015-no-virtiofs-hot-caches.md @@ -1,4 +1,4 @@ -# 015: No steady-state virtiofs for churning caches +# 015: No virtiofs for churning caches **Date:** 2026-04-27 **Status:** accepted @@ -9,34 +9,25 @@ ## Decision -Plugins MUST NOT use virtiofs as the **steady-state runtime path** for churning data — package-manager caches, build-artifact caches, indexer state. Runtime cache I/O lives on `root.img`, which persists per workspace, at the tool's default location (or via env-var redirect from a setup script when the default is not under `$HOME`). +Plugins MUST NOT virtiofs-mount churning data — package-manager caches, build-artifact caches, indexer state. Cache I/O lives on `root.img`, which persists per workspace, at the tool's default location (or via env-var redirect from a setup script when the default is not under `$HOME`). -Use virtiofs only where in-place cross-boundary semantics matter (e.g. source trees). - -### Bootstrap exception - -A read-only virtiofs mount used for a one-shot copy into `root.img` during the setup script is permitted. The FD high-water mark is bounded by the file count copied (one-time peak), not by session length. +If the host has a useful warm cache to seed the guest with, plugins SHOULD deliver it via a host-side **plugin command** (`plugins//commands/.sh`) invoked from a `post-up` hook. The command streams the cache into the guest over the existing SSH channel (e.g. `tar | nixbox run "tar -x"`) and writes a sentinel for idempotency. No virtiofs mount is involved. -Bootstrap mounts MUST: -- Be `readonly = true`. -- Mount at a side path (e.g. `/mnt/host-cache/coursier`), never at the tool's runtime cache location. -- Be read by an idempotent setup script that writes a sentinel and skips re-runs. -- Not be touched after warmup (no live host↔guest cache sharing). - -Bootstrap mounts SHOULD be released after warmup via a `post-up` hook (`nixbox unmount ` is idempotent and frees the virtiofsd FDs accumulated during the copy). **In-tree plugins MUST release.** Third-party plugins are encouraged to release but are not required to — the FD high-water from a single bootstrap is well under the 524288 ceiling. +Use virtiofs only where in-place cross-boundary semantics matter (e.g. source trees). ## Rationale -`virtiofsd --cache=auto` holds a backing-file FD per served file and never meaningfully drops them — the FD set grows monotonically with workload. A persistent mount of a churning cache fails on this alone; the constrained atomic-op semantics noted in ADR-001 are an aggravating secondary cost. Bootstrap mounts pay the FD cost once (bounded by file count), then go quiet. +`virtiofsd --cache=auto` holds a backing-file FD per served file and never meaningfully drops them — the FD set grows monotonically with workload. A persistent mount of a churning cache fails on this alone; the constrained atomic-op semantics noted in ADR-001 are an aggravating secondary cost. + +A plugin command driven by `post-up` is the right primitive for one-shot host→guest data transfer: it runs host-side where the data is, has the same SSH channel `bin/nixbox` uses internally, and is also user-invocable for manual refreshes (mirroring the established `nixbox aws login` and `nixbox claude-code sync-config` patterns). ## Consequences - New plugins do not declare runtime mounts for cache paths. Defaults under `$HOME` work; non-`$HOME` defaults get an env-var redirect from `scripts/setup.sh` (plugins cannot inject `env`, per ADR-013). -- Plugins MAY declare RO bootstrap mounts at side paths and rsync from them in setup. -- Live host↔guest cache sharing is gone; first boot per workspace warms from host, subsequent boots use `root.img` directly. -- `plugins/scala-sbt` adopts the bootstrap pattern: RO mounts at `/mnt/host-cache/{coursier,ivy2}`, rsynced into `~/.cache/coursier` and `~/.ivy2` on first boot, sentinel-guarded; mounts released via `post-up` hook. Adds `rsync` to its packages. -- `nixbox unmount` is idempotent (no-op + log if the target is not mounted), so post-up hooks are safe to run unconditionally on every `up`. +- Host→guest cache seeding lives in a plugin command, called from `hooks.post-up`. The command is responsible for idempotency. +- Live host↔guest cache sharing is gone; the warmup snapshots host state at first boot, then guest and host diverge. +- `plugins/scala-sbt`: warmup delivered by `nixbox scala-sbt warm-cache` (host-side `tar` piped into the guest). No virtiofs mounts at all in this plugin. ## Future work -VM snapshotting would replace the bootstrap rsync with a pre-warmed image; tracked separately. +VM snapshotting would replace the per-workspace warmup with a pre-warmed image; tracked separately. diff --git a/plugins/scala-sbt/README.md b/plugins/scala-sbt/README.md index ad33bbf..c1cdf9d 100644 --- a/plugins/scala-sbt/README.md +++ b/plugins/scala-sbt/README.md @@ -12,22 +12,25 @@ Scala toolchain with optional private Maven/Nexus credentials. | Category | Details | |---|---| -| **Packages** | `sbt`, `scala`, `scala-cli`, `rsync` | +| **Packages** | `sbt`, `scala`, `scala-cli` | | **Domains** | `maven.org`, `scala-sbt.org`, plus `MAVEN_REPO_HOST` if set | +| **Commands** | `nixbox scala-sbt warm-cache` | ## Caches Coursier and ivy2 caches live on the guest's `root.img` at default paths -(`~/.cache/coursier`, `~/.ivy2`). On first boot per workspace, the plugin -warms them by `rsync`-copying from the host's `~/.cache/coursier` and -`~/.ivy2` (mounted read-only at `/mnt/host-cache/...`). The mounts are -released via `post-up` hook once warmup completes — `virtiofsd` exits and -the runtime sbt I/O lives entirely on `root.img`. +(`~/.cache/coursier`, `~/.ivy2`). On `nixbox up` the plugin's `post-up` +hook runs `nixbox scala-sbt warm-cache`, which streams the host's +`~/.cache/coursier` and `~/.ivy2` over the SSH channel into the guest +(idempotent — guarded by a sentinel inside each cache dir). Subsequent +boots are no-ops. Runtime sbt I/O lives entirely on `root.img` and never +crosses virtiofs (see +[ADR-015](../../docs/decisions/015-no-virtiofs-hot-caches.md)). If the host paths don't exist, the warmup is skipped and the cache is -populated by `sbt update` over the network. See -[ADR-015](../../docs/decisions/015-no-virtiofs-hot-caches.md) for the -rationale. +populated by `sbt update` over the network. You can also re-run the +command manually (`nixbox scala-sbt warm-cache`) to refresh after +removing the sentinel files. ## Private repository credentials diff --git a/plugins/scala-sbt/commands/warm-cache.sh b/plugins/scala-sbt/commands/warm-cache.sh new file mode 100755 index 0000000..39a08fd --- /dev/null +++ b/plugins/scala-sbt/commands/warm-cache.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Plugin command: nixbox scala-sbt warm-cache +# One-shot copy of the host's coursier and ivy2 caches into the guest's +# ~/.cache/coursier and ~/.ivy2. Idempotent — guarded by a sentinel file +# inside each guest cache dir. Invoked from the plugin's post-up hook so +# warmup runs automatically once per workspace. + +die() { printf '\r%s\n' "ERROR: $*" >&2; exit 1; } +log() { printf '\r%s\n' "$*"; } +log_sub() { printf '\r %s\n' "$*"; } + +warm() { + local host_src="$1" guest_relpath="$2" name="$3" + [ -d "$host_src" ] || { log_sub "skip $name: $host_src not found on host"; return 0; } + + if nixbox run "test -f \"\$HOME/$guest_relpath/.nixbox-warmed\"" 2>/dev/null; then + log_sub "$name: already warmed" + return 0 + fi + + log "==> Warming $name cache from host (one-time per workspace)..." + nixbox run "mkdir -p \"\$HOME/$guest_relpath\"" + tar -C "$host_src" -cf - . \ + | nixbox run "tar -C \"\$HOME/$guest_relpath\" -xf - && touch \"\$HOME/$guest_relpath/.nixbox-warmed\"" + log_sub "$name: done" +} + +warm "$HOME/.cache/coursier" ".cache/coursier" coursier +warm "$HOME/.ivy2" ".ivy2" ivy2 diff --git a/plugins/scala-sbt/default.nix b/plugins/scala-sbt/default.nix index b3bc01a..969339f 100644 --- a/plugins/scala-sbt/default.nix +++ b/plugins/scala-sbt/default.nix @@ -1,25 +1,10 @@ -let - home = builtins.getEnv "HOME"; - # Bootstrap mount: RO virtiofs share read once at boot by setup.sh, then - # idle for the rest of the VM's lifetime (see ADR-015). Skipped if the - # host path doesn't exist. - warmFromHost = src: target: - if home != "" && builtins.pathExists (home + src) - then [ { source = "~" + src; inherit target; readonly = true; } ] - else [ ]; -in { nix.packages = [ "sbt" "scala" "scala-cli" - "rsync" ]; - mounts = - warmFromHost "/.cache/coursier" "/mnt/host-cache/coursier" - ++ warmFromHost "/.ivy2" "/mnt/host-cache/ivy2"; - network.domains = [ "maven.org" @@ -32,10 +17,7 @@ in scripts = [ ./scripts/setup.sh ]; - # Release bootstrap mounts after warmup (ADR-015): drops the guest mount, - # detaches the virtio device, kills virtiofsd → frees its accumulated FDs. - hooks.post-up = [ - "nixbox unmount /mnt/host-cache/coursier" - "nixbox unmount /mnt/host-cache/ivy2" - ]; + # Warm coursier/ivy2 caches from host on every up. The command is + # sentinel-guarded inside the guest, so re-runs are no-ops once warmed. + hooks.post-up = [ "nixbox scala-sbt warm-cache" ]; } diff --git a/plugins/scala-sbt/scripts/setup.sh b/plugins/scala-sbt/scripts/setup.sh index 90a76ba..398bf8c 100755 --- a/plugins/scala-sbt/scripts/setup.sh +++ b/plugins/scala-sbt/scripts/setup.sh @@ -11,18 +11,3 @@ user=$MAVEN_REPO_USER password=$MAVEN_REPO_PASSWORD EOF fi - -# One-shot warmup of host caches into root.img on first boot per workspace. -# Runtime sbt I/O lives entirely on root.img (see ADR-015). -warm_cache() { - local src="$1" dst="$2" name="$3" - [ -d "$src" ] || return 0 - [ -f "$dst/.nixbox-warmed" ] && return 0 - echo "==> Warming $name cache from host (one-time per workspace)..." - mkdir -p "$dst" - rsync -a "$src/" "$dst/" - touch "$dst/.nixbox-warmed" -} - -warm_cache /mnt/host-cache/coursier "$HOME/.cache/coursier" coursier -warm_cache /mnt/host-cache/ivy2 "$HOME/.ivy2" ivy2 From 6e5298a9ab1f5f83389723c2236c80f6106d72ec Mon Sep 17 00:00:00 2001 From: Razvan Laurus Date: Mon, 27 Apr 2026 11:12:34 +0200 Subject: [PATCH 08/11] docs(adr-015): soften from project-wide rule to in-tree policy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reframe the ADR so it documents trade-offs rather than dictating to third-party plugin authors: - Title: "No virtiofs ..." → "Avoid virtiofs ..." - Decision: "Plugins MUST NOT ..." → "In-tree plugins do not ..." - Add an explicit paragraph stating that third-party plugins are not bound; authors who virtiofs-mount churning caches accept the FD-pressure cost documented in #18 - Plugin-command pattern: presented as one clean alternative, not "the right primitive" - Consequences scoped to in-tree plugins The substance — what the project does and why, and what it costs to do otherwise — is unchanged. Only the framing softens. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/decisions/015-no-virtiofs-hot-caches.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/decisions/015-no-virtiofs-hot-caches.md b/docs/decisions/015-no-virtiofs-hot-caches.md index 04944c0..6b66109 100644 --- a/docs/decisions/015-no-virtiofs-hot-caches.md +++ b/docs/decisions/015-no-virtiofs-hot-caches.md @@ -1,4 +1,4 @@ -# 015: No virtiofs for churning caches +# 015: Avoid virtiofs for churning caches **Date:** 2026-04-27 **Status:** accepted @@ -9,9 +9,11 @@ ## Decision -Plugins MUST NOT virtiofs-mount churning data — package-manager caches, build-artifact caches, indexer state. Cache I/O lives on `root.img`, which persists per workspace, at the tool's default location (or via env-var redirect from a setup script when the default is not under `$HOME`). +In-tree plugins do not virtiofs-mount churning data — package-manager caches, build-artifact caches, indexer state. Cache I/O lives on `root.img`, which persists per workspace, at the tool's default location (or via env-var redirect from a setup script when the default is not under `$HOME`). -If the host has a useful warm cache to seed the guest with, plugins SHOULD deliver it via a host-side **plugin command** (`plugins//commands/.sh`) invoked from a `post-up` hook. The command streams the cache into the guest over the existing SSH channel (e.g. `tar | nixbox run "tar -x"`) and writes a sentinel for idempotency. No virtiofs mount is involved. +When the host has a useful warm cache to seed the guest from, in-tree plugins deliver it via a host-side **plugin command** (`plugins//commands/.sh`) invoked from a `post-up` hook — the pattern established by `nixbox aws login` and `nixbox claude-code sync-config`. The command streams the cache into the guest over the existing SSH channel (e.g. `tar | nixbox run "tar -x"`) and writes a sentinel for idempotency. No virtiofs mount is involved. + +Third-party plugins are not bound by this. Authors who virtiofs-mount churning caches accept the FD-pressure cost documented in #18: `virtiofsd --cache=auto` accumulates FDs monotonically, so any session long enough will exhaust the ceiling. The plugin-command pattern above is one alternative, not the only one. Use virtiofs only where in-place cross-boundary semantics matter (e.g. source trees). @@ -19,13 +21,13 @@ Use virtiofs only where in-place cross-boundary semantics matter (e.g. source tr `virtiofsd --cache=auto` holds a backing-file FD per served file and never meaningfully drops them — the FD set grows monotonically with workload. A persistent mount of a churning cache fails on this alone; the constrained atomic-op semantics noted in ADR-001 are an aggravating secondary cost. -A plugin command driven by `post-up` is the right primitive for one-shot host→guest data transfer: it runs host-side where the data is, has the same SSH channel `bin/nixbox` uses internally, and is also user-invocable for manual refreshes (mirroring the established `nixbox aws login` and `nixbox claude-code sync-config` patterns). +A plugin command driven by `post-up` is one clean alternative for in-tree plugins that need to seed a guest cache: it runs host-side where the data is, uses the SSH channel `bin/nixbox` already has, and is user-invocable for manual refreshes. It mirrors the existing `nixbox aws login` and `nixbox claude-code sync-config` patterns. ## Consequences -- New plugins do not declare runtime mounts for cache paths. Defaults under `$HOME` work; non-`$HOME` defaults get an env-var redirect from `scripts/setup.sh` (plugins cannot inject `env`, per ADR-013). -- Host→guest cache seeding lives in a plugin command, called from `hooks.post-up`. The command is responsible for idempotency. -- Live host↔guest cache sharing is gone; the warmup snapshots host state at first boot, then guest and host diverge. +- New in-tree plugins do not declare runtime mounts for cache paths. Defaults under `$HOME` work; non-`$HOME` defaults get an env-var redirect from `scripts/setup.sh` (plugins cannot inject `env`, per ADR-013). +- For in-tree plugins, host→guest cache seeding lives in a plugin command called from `hooks.post-up`. The command is responsible for idempotency. +- Live host↔guest cache sharing is gone for in-tree plugins; the warmup snapshots host state at first boot, then guest and host diverge. - `plugins/scala-sbt`: warmup delivered by `nixbox scala-sbt warm-cache` (host-side `tar` piped into the guest). No virtiofs mounts at all in this plugin. ## Future work From abf9ac279b3423e6a0d85dca7c6693064c43e2b6 Mon Sep 17 00:00:00 2001 From: Razvan Laurus Date: Mon, 27 Apr 2026 11:39:54 +0200 Subject: [PATCH 09/11] docs(adr-015): trim further MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cut the Rationale section entirely — its FD-monotonic argument is already in Problem; the plugin-command motivation is conveyed by the "established pattern" cue in Decision. Compress Consequences to two bullets. 36 → 27 lines. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/decisions/015-no-virtiofs-hot-caches.md | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/docs/decisions/015-no-virtiofs-hot-caches.md b/docs/decisions/015-no-virtiofs-hot-caches.md index 6b66109..fa0d02e 100644 --- a/docs/decisions/015-no-virtiofs-hot-caches.md +++ b/docs/decisions/015-no-virtiofs-hot-caches.md @@ -5,30 +5,22 @@ ## Problem -`virtiofsd --cache=auto` accumulates backing-file FDs monotonically; under FD-heavy workloads (e.g. `sbt update` validating thousands of `.jar__sha1` files) host shares pin at the process `RLIMIT_NOFILE` ceiling and surface as `ENFILE` in the guest (#18). Raising the limit (PR #20, 524288) buys headroom but doesn't change the monotonicity. +`virtiofsd --cache=auto` accumulates backing-file FDs monotonically. Under churning workloads (`sbt update`, package-manager refreshes), shares pin at `RLIMIT_NOFILE` and surface as `ENFILE` in the guest (#18). Raising the ceiling (PR #20) doesn't change the trajectory. ## Decision -In-tree plugins do not virtiofs-mount churning data — package-manager caches, build-artifact caches, indexer state. Cache I/O lives on `root.img`, which persists per workspace, at the tool's default location (or via env-var redirect from a setup script when the default is not under `$HOME`). +In-tree plugins do not virtiofs-mount churning data — package-manager caches, build-artifact caches, indexer state. Cache I/O lives on `root.img` (persists per workspace) at the tool's default location, or via env-var redirect from a setup script. -When the host has a useful warm cache to seed the guest from, in-tree plugins deliver it via a host-side **plugin command** (`plugins//commands/.sh`) invoked from a `post-up` hook — the pattern established by `nixbox aws login` and `nixbox claude-code sync-config`. The command streams the cache into the guest over the existing SSH channel (e.g. `tar | nixbox run "tar -x"`) and writes a sentinel for idempotency. No virtiofs mount is involved. +When useful host warm-cache state exists, in-tree plugins deliver it via a host-side **plugin command** invoked from `post-up` — the pattern of `nixbox aws login` and `nixbox claude-code sync-config`. Streams over the existing SSH channel (`tar | nixbox run "tar -x"`); sentinel-guarded. -Third-party plugins are not bound by this. Authors who virtiofs-mount churning caches accept the FD-pressure cost documented in #18: `virtiofsd --cache=auto` accumulates FDs monotonically, so any session long enough will exhaust the ceiling. The plugin-command pattern above is one alternative, not the only one. +Third-party plugins are not bound. Authors who virtiofs-mount churning caches accept the FD cost — any long enough session will exhaust the ceiling. The plugin-command pattern is one alternative, not the only one. Use virtiofs only where in-place cross-boundary semantics matter (e.g. source trees). -## Rationale - -`virtiofsd --cache=auto` holds a backing-file FD per served file and never meaningfully drops them — the FD set grows monotonically with workload. A persistent mount of a churning cache fails on this alone; the constrained atomic-op semantics noted in ADR-001 are an aggravating secondary cost. - -A plugin command driven by `post-up` is one clean alternative for in-tree plugins that need to seed a guest cache: it runs host-side where the data is, uses the SSH channel `bin/nixbox` already has, and is user-invocable for manual refreshes. It mirrors the existing `nixbox aws login` and `nixbox claude-code sync-config` patterns. - ## Consequences -- New in-tree plugins do not declare runtime mounts for cache paths. Defaults under `$HOME` work; non-`$HOME` defaults get an env-var redirect from `scripts/setup.sh` (plugins cannot inject `env`, per ADR-013). -- For in-tree plugins, host→guest cache seeding lives in a plugin command called from `hooks.post-up`. The command is responsible for idempotency. -- Live host↔guest cache sharing is gone for in-tree plugins; the warmup snapshots host state at first boot, then guest and host diverge. -- `plugins/scala-sbt`: warmup delivered by `nixbox scala-sbt warm-cache` (host-side `tar` piped into the guest). No virtiofs mounts at all in this plugin. +- `plugins/scala-sbt`: warmup via `nixbox scala-sbt warm-cache`. No virtiofs mounts. +- Host↔guest cache sharing is one-shot, not live: warmup snapshots host state at first boot, then guest and host diverge. ## Future work From 47ebf66bb152d267ad9d81d55cee3359481fbdfa Mon Sep 17 00:00:00 2001 From: Razvan Laurus Date: Mon, 27 Apr 2026 12:04:39 +0200 Subject: [PATCH 10/11] docs(adr-015): present VM snapshotting as possibility, not future work "Future work" implied roadmap commitment. Reframe as a possible direction the trade-off could be revisited from, with explicit "not committed work" framing. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/decisions/015-no-virtiofs-hot-caches.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/decisions/015-no-virtiofs-hot-caches.md b/docs/decisions/015-no-virtiofs-hot-caches.md index fa0d02e..7fc0ad2 100644 --- a/docs/decisions/015-no-virtiofs-hot-caches.md +++ b/docs/decisions/015-no-virtiofs-hot-caches.md @@ -22,6 +22,6 @@ Use virtiofs only where in-place cross-boundary semantics matter (e.g. source tr - `plugins/scala-sbt`: warmup via `nixbox scala-sbt warm-cache`. No virtiofs mounts. - Host↔guest cache sharing is one-shot, not live: warmup snapshots host state at first boot, then guest and host diverge. -## Future work +## Possible direction -VM snapshotting would replace the per-workspace warmup with a pre-warmed image; tracked separately. +VM snapshotting (qcow2 backing or cloud-hypervisor save/restore) could replace the per-workspace warmup with a pre-warmed image. Not committed work — noted as one way this trade-off might be revisited. From c4f580342bd2c63a0870c03be8861c430e095c39 Mon Sep 17 00:00:00 2001 From: Razvan Laurus Date: Mon, 27 Apr 2026 12:07:39 +0200 Subject: [PATCH 11/11] docs(readme): refresh scala-sbt section; note cache choice in limitations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - scala-sbt provides line: include scala-cli (added in #15) and the warm-cache plugin command. Mention root.img persistence + ADR-015 link in the trailing paragraph. - Known limitations: add "No live host↔guest cache sharing" entry pointing at ADR-015 — useful for users who'd otherwise expect virtiofs-style mount semantics for caches. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ce00617..38453f0 100644 --- a/README.md +++ b/README.md @@ -229,9 +229,9 @@ Authenticates via SSO, injects temporary credentials into the VM, and logs into ### scala-sbt -**Provides:** `sbt` + `scala` packages, Maven/SBT resolver domains. +**Provides:** `sbt`, `scala`, `scala-cli` packages, Maven/SBT resolver domains, and `nixbox scala-sbt warm-cache` (auto-runs on `up`). -The JDK is specified separately via `nix.packages` so you control the version. Set `MAVEN_REPO_HOST`, `MAVEN_REPO_USER`, and `MAVEN_REPO_PASSWORD` env vars to auto-generate `~/.sbt/1.0/credentials.sbt` for private repositories. +The JDK is specified separately via `nix.packages` so you control the version. Set `MAVEN_REPO_HOST`, `MAVEN_REPO_USER`, and `MAVEN_REPO_PASSWORD` env vars to auto-generate `~/.sbt/1.0/credentials.sbt` for private repositories. Coursier and ivy2 caches live on `root.img` (per-workspace persistent), seeded from host on first boot — see [ADR 015](docs/decisions/015-no-virtiofs-hot-caches.md). **Example config:** @@ -256,6 +256,7 @@ The JDK is specified separately via `nix.packages` so you control the version. S - **Concurrent VMs** — up to 64 concurrent VMs supported, each with per-VM network isolation via slot-based IP allocation. - **virtiofs + `O_TMPFILE`** — virtiofs does not support `O_TMPFILE`. Tools that hit this (e.g. Node.js/Claude Code) need tmpfs overlays on affected dirs — the `claude-code` plugin handles this automatically. +- **No live host↔guest cache sharing** — package-manager caches (coursier, ivy2, etc.) live on `root.img` per workspace, not virtiofs-shared with host. In-tree plugins may seed from host on first boot (e.g. `nixbox scala-sbt warm-cache`); after that, host and guest diverge. See [ADR 015](docs/decisions/015-no-virtiofs-hot-caches.md). - **Claude Code conversations** — Claude Code stores conversations under `~/.claude/projects/` keyed by the workspace's absolute path. Since the workspace path differs between host and guest (e.g. `/home/you/workspace` vs `/home/vmuser/workspace`), conversations don't carry over between the two. Workaround: symlink the guest-side conversation directory to the host's. ## Acknowledgments