diff --git a/README.md b/README.md index ce00617..38453f0 100644 --- a/README.md +++ b/README.md @@ -229,9 +229,9 @@ Authenticates via SSO, injects temporary credentials into the VM, and logs into ### scala-sbt -**Provides:** `sbt` + `scala` packages, Maven/SBT resolver domains. +**Provides:** `sbt`, `scala`, `scala-cli` packages, Maven/SBT resolver domains, and `nixbox scala-sbt warm-cache` (auto-runs on `up`). -The JDK is specified separately via `nix.packages` so you control the version. Set `MAVEN_REPO_HOST`, `MAVEN_REPO_USER`, and `MAVEN_REPO_PASSWORD` env vars to auto-generate `~/.sbt/1.0/credentials.sbt` for private repositories. +The JDK is specified separately via `nix.packages` so you control the version. Set `MAVEN_REPO_HOST`, `MAVEN_REPO_USER`, and `MAVEN_REPO_PASSWORD` env vars to auto-generate `~/.sbt/1.0/credentials.sbt` for private repositories. Coursier and ivy2 caches live on `root.img` (per-workspace persistent), seeded from host on first boot — see [ADR 015](docs/decisions/015-no-virtiofs-hot-caches.md). **Example config:** @@ -256,6 +256,7 @@ The JDK is specified separately via `nix.packages` so you control the version. S - **Concurrent VMs** — up to 64 concurrent VMs supported, each with per-VM network isolation via slot-based IP allocation. - **virtiofs + `O_TMPFILE`** — virtiofs does not support `O_TMPFILE`. Tools that hit this (e.g. Node.js/Claude Code) need tmpfs overlays on affected dirs — the `claude-code` plugin handles this automatically. +- **No live host↔guest cache sharing** — package-manager caches (coursier, ivy2, etc.) live on `root.img` per workspace, not virtiofs-shared with host. In-tree plugins may seed from host on first boot (e.g. `nixbox scala-sbt warm-cache`); after that, host and guest diverge. See [ADR 015](docs/decisions/015-no-virtiofs-hot-caches.md). - **Claude Code conversations** — Claude Code stores conversations under `~/.claude/projects/` keyed by the workspace's absolute path. Since the workspace path differs between host and guest (e.g. `/home/you/workspace` vs `/home/vmuser/workspace`), conversations don't carry over between the two. Workaround: symlink the guest-side conversation directory to the host's. ## Acknowledgments diff --git a/docs/decisions/015-no-virtiofs-hot-caches.md b/docs/decisions/015-no-virtiofs-hot-caches.md new file mode 100644 index 0000000..7fc0ad2 --- /dev/null +++ b/docs/decisions/015-no-virtiofs-hot-caches.md @@ -0,0 +1,27 @@ +# 015: Avoid virtiofs for churning caches + +**Date:** 2026-04-27 +**Status:** accepted + +## Problem + +`virtiofsd --cache=auto` accumulates backing-file FDs monotonically. Under churning workloads (`sbt update`, package-manager refreshes), shares pin at `RLIMIT_NOFILE` and surface as `ENFILE` in the guest (#18). Raising the ceiling (PR #20) doesn't change the trajectory. + +## Decision + +In-tree plugins do not virtiofs-mount churning data — package-manager caches, build-artifact caches, indexer state. Cache I/O lives on `root.img` (persists per workspace) at the tool's default location, or via env-var redirect from a setup script. + +When useful host warm-cache state exists, in-tree plugins deliver it via a host-side **plugin command** invoked from `post-up` — the pattern of `nixbox aws login` and `nixbox claude-code sync-config`. Streams over the existing SSH channel (`tar | nixbox run "tar -x"`); sentinel-guarded. + +Third-party plugins are not bound. Authors who virtiofs-mount churning caches accept the FD cost — any long enough session will exhaust the ceiling. The plugin-command pattern is one alternative, not the only one. + +Use virtiofs only where in-place cross-boundary semantics matter (e.g. source trees). + +## Consequences + +- `plugins/scala-sbt`: warmup via `nixbox scala-sbt warm-cache`. No virtiofs mounts. +- Host↔guest cache sharing is one-shot, not live: warmup snapshots host state at first boot, then guest and host diverge. + +## Possible direction + +VM snapshotting (qcow2 backing or cloud-hypervisor save/restore) could replace the per-workspace warmup with a pre-warmed image. Not committed work — noted as one way this trade-off might be revisited. diff --git a/plugins/scala-sbt/README.md b/plugins/scala-sbt/README.md index 030b631..c1cdf9d 100644 --- a/plugins/scala-sbt/README.md +++ b/plugins/scala-sbt/README.md @@ -12,9 +12,25 @@ Scala toolchain with optional private Maven/Nexus credentials. | Category | Details | |---|---| -| **Packages** | `sbt`, `scala` | -| **Mounts** | `~/.cache/coursier`, `~/.ivy2` (only if they exist on host) | +| **Packages** | `sbt`, `scala`, `scala-cli` | | **Domains** | `maven.org`, `scala-sbt.org`, plus `MAVEN_REPO_HOST` if set | +| **Commands** | `nixbox scala-sbt warm-cache` | + +## Caches + +Coursier and ivy2 caches live on the guest's `root.img` at default paths +(`~/.cache/coursier`, `~/.ivy2`). On `nixbox up` the plugin's `post-up` +hook runs `nixbox scala-sbt warm-cache`, which streams the host's +`~/.cache/coursier` and `~/.ivy2` over the SSH channel into the guest +(idempotent — guarded by a sentinel inside each cache dir). Subsequent +boots are no-ops. Runtime sbt I/O lives entirely on `root.img` and never +crosses virtiofs (see +[ADR-015](../../docs/decisions/015-no-virtiofs-hot-caches.md)). + +If the host paths don't exist, the warmup is skipped and the cache is +populated by `sbt update` over the network. You can also re-run the +command manually (`nixbox scala-sbt warm-cache`) to refresh after +removing the sentinel files. ## Private repository credentials diff --git a/plugins/scala-sbt/commands/warm-cache.sh b/plugins/scala-sbt/commands/warm-cache.sh new file mode 100755 index 0000000..39a08fd --- /dev/null +++ b/plugins/scala-sbt/commands/warm-cache.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Plugin command: nixbox scala-sbt warm-cache +# One-shot copy of the host's coursier and ivy2 caches into the guest's +# ~/.cache/coursier and ~/.ivy2. Idempotent — guarded by a sentinel file +# inside each guest cache dir. Invoked from the plugin's post-up hook so +# warmup runs automatically once per workspace. + +die() { printf '\r%s\n' "ERROR: $*" >&2; exit 1; } +log() { printf '\r%s\n' "$*"; } +log_sub() { printf '\r %s\n' "$*"; } + +warm() { + local host_src="$1" guest_relpath="$2" name="$3" + [ -d "$host_src" ] || { log_sub "skip $name: $host_src not found on host"; return 0; } + + if nixbox run "test -f \"\$HOME/$guest_relpath/.nixbox-warmed\"" 2>/dev/null; then + log_sub "$name: already warmed" + return 0 + fi + + log "==> Warming $name cache from host (one-time per workspace)..." + nixbox run "mkdir -p \"\$HOME/$guest_relpath\"" + tar -C "$host_src" -cf - . \ + | nixbox run "tar -C \"\$HOME/$guest_relpath\" -xf - && touch \"\$HOME/$guest_relpath/.nixbox-warmed\"" + log_sub "$name: done" +} + +warm "$HOME/.cache/coursier" ".cache/coursier" coursier +warm "$HOME/.ivy2" ".ivy2" ivy2 diff --git a/plugins/scala-sbt/default.nix b/plugins/scala-sbt/default.nix index ad92d47..969339f 100644 --- a/plugins/scala-sbt/default.nix +++ b/plugins/scala-sbt/default.nix @@ -1,10 +1,3 @@ -let - home = builtins.getEnv "HOME"; - mountIf = path: target: - if home != "" && builtins.pathExists (home + path) - then [ { source = "~" + path; inherit target; } ] - else [ ]; -in { nix.packages = [ "sbt" @@ -12,10 +5,6 @@ in "scala-cli" ]; - mounts = - mountIf "/.cache/coursier" "~/.cache/coursier" - ++ mountIf "/.ivy2" "~/.ivy2"; - network.domains = [ "maven.org" @@ -27,4 +16,8 @@ in ); scripts = [ ./scripts/setup.sh ]; + + # Warm coursier/ivy2 caches from host on every up. The command is + # sentinel-guarded inside the guest, so re-runs are no-ops once warmed. + hooks.post-up = [ "nixbox scala-sbt warm-cache" ]; }