Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,9 +229,9 @@ Authenticates via SSO, injects temporary credentials into the VM, and logs into

### scala-sbt

**Provides:** `sbt` + `scala` packages, Maven/SBT resolver domains.
**Provides:** `sbt`, `scala`, `scala-cli` packages, Maven/SBT resolver domains, and `nixbox scala-sbt warm-cache` (auto-runs on `up`).

The JDK is specified separately via `nix.packages` so you control the version. Set `MAVEN_REPO_HOST`, `MAVEN_REPO_USER`, and `MAVEN_REPO_PASSWORD` env vars to auto-generate `~/.sbt/1.0/credentials.sbt` for private repositories.
The JDK is specified separately via `nix.packages` so you control the version. Set `MAVEN_REPO_HOST`, `MAVEN_REPO_USER`, and `MAVEN_REPO_PASSWORD` env vars to auto-generate `~/.sbt/1.0/credentials.sbt` for private repositories. Coursier and ivy2 caches live on `root.img` (per-workspace persistent), seeded from host on first boot — see [ADR 015](docs/decisions/015-no-virtiofs-hot-caches.md).

**Example config:**

Expand All @@ -256,6 +256,7 @@ The JDK is specified separately via `nix.packages` so you control the version. S

- **Concurrent VMs** — up to 64 concurrent VMs supported, each with per-VM network isolation via slot-based IP allocation.
- **virtiofs + `O_TMPFILE`** — virtiofs does not support `O_TMPFILE`. Tools that hit this (e.g. Node.js/Claude Code) need tmpfs overlays on affected dirs — the `claude-code` plugin handles this automatically.
- **No live host↔guest cache sharing** — package-manager caches (coursier, ivy2, etc.) live on `root.img` per workspace, not virtiofs-shared with host. In-tree plugins may seed from host on first boot (e.g. `nixbox scala-sbt warm-cache`); after that, host and guest diverge. See [ADR 015](docs/decisions/015-no-virtiofs-hot-caches.md).
- **Claude Code conversations** — Claude Code stores conversations under `~/.claude/projects/` keyed by the workspace's absolute path. Since the workspace path differs between host and guest (e.g. `/home/you/workspace` vs `/home/vmuser/workspace`), conversations don't carry over between the two. Workaround: symlink the guest-side conversation directory to the host's.

## Acknowledgments
Expand Down
27 changes: 27 additions & 0 deletions docs/decisions/015-no-virtiofs-hot-caches.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# 015: Avoid virtiofs for churning caches

**Date:** 2026-04-27
**Status:** accepted

## Problem

`virtiofsd --cache=auto` accumulates backing-file FDs monotonically. Under churning workloads (`sbt update`, package-manager refreshes), shares pin at `RLIMIT_NOFILE` and surface as `ENFILE` in the guest (#18). Raising the ceiling (PR #20) doesn't change the trajectory.

## Decision

In-tree plugins do not virtiofs-mount churning data — package-manager caches, build-artifact caches, indexer state. Cache I/O lives on `root.img` (persists per workspace) at the tool's default location, or via env-var redirect from a setup script.

When useful host warm-cache state exists, in-tree plugins deliver it via a host-side **plugin command** invoked from `post-up` — the pattern of `nixbox aws login` and `nixbox claude-code sync-config`. Streams over the existing SSH channel (`tar | nixbox run "tar -x"`); sentinel-guarded.

Third-party plugins are not bound. Authors who virtiofs-mount churning caches accept the FD cost — any long enough session will exhaust the ceiling. The plugin-command pattern is one alternative, not the only one.

Use virtiofs only where in-place cross-boundary semantics matter (e.g. source trees).

## Consequences

- `plugins/scala-sbt`: warmup via `nixbox scala-sbt warm-cache`. No virtiofs mounts.
- Host↔guest cache sharing is one-shot, not live: warmup snapshots host state at first boot, then guest and host diverge.

## Possible direction

VM snapshotting (qcow2 backing or cloud-hypervisor save/restore) could replace the per-workspace warmup with a pre-warmed image. Not committed work — noted as one way this trade-off might be revisited.
20 changes: 18 additions & 2 deletions plugins/scala-sbt/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,25 @@ Scala toolchain with optional private Maven/Nexus credentials.

| Category | Details |
|---|---|
| **Packages** | `sbt`, `scala` |
| **Mounts** | `~/.cache/coursier`, `~/.ivy2` (only if they exist on host) |
| **Packages** | `sbt`, `scala`, `scala-cli` |
| **Domains** | `maven.org`, `scala-sbt.org`, plus `MAVEN_REPO_HOST` if set |
| **Commands** | `nixbox scala-sbt warm-cache` |

## Caches

Coursier and ivy2 caches live on the guest's `root.img` at default paths
(`~/.cache/coursier`, `~/.ivy2`). On `nixbox up` the plugin's `post-up`
hook runs `nixbox scala-sbt warm-cache`, which streams the host's
`~/.cache/coursier` and `~/.ivy2` over the SSH channel into the guest
(idempotent — guarded by a sentinel inside each cache dir). Subsequent
boots are no-ops. Runtime sbt I/O lives entirely on `root.img` and never
crosses virtiofs (see
[ADR-015](../../docs/decisions/015-no-virtiofs-hot-caches.md)).

If the host paths don't exist, the warmup is skipped and the cache is
populated by `sbt update` over the network. You can also re-run the
command manually (`nixbox scala-sbt warm-cache`) to refresh after
removing the sentinel files.

## Private repository credentials

Expand Down
31 changes: 31 additions & 0 deletions plugins/scala-sbt/commands/warm-cache.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
set -euo pipefail

# Plugin command: nixbox scala-sbt warm-cache
# One-shot copy of the host's coursier and ivy2 caches into the guest's
# ~/.cache/coursier and ~/.ivy2. Idempotent — guarded by a sentinel file
# inside each guest cache dir. Invoked from the plugin's post-up hook so
# warmup runs automatically once per workspace.

die() { printf '\r%s\n' "ERROR: $*" >&2; exit 1; }
log() { printf '\r%s\n' "$*"; }
log_sub() { printf '\r %s\n' "$*"; }

warm() {
local host_src="$1" guest_relpath="$2" name="$3"
[ -d "$host_src" ] || { log_sub "skip $name: $host_src not found on host"; return 0; }

if nixbox run "test -f \"\$HOME/$guest_relpath/.nixbox-warmed\"" 2>/dev/null; then
log_sub "$name: already warmed"
return 0
fi

log "==> Warming $name cache from host (one-time per workspace)..."
nixbox run "mkdir -p \"\$HOME/$guest_relpath\""
tar -C "$host_src" -cf - . \
| nixbox run "tar -C \"\$HOME/$guest_relpath\" -xf - && touch \"\$HOME/$guest_relpath/.nixbox-warmed\""
log_sub "$name: done"
Comment thread
razvanz marked this conversation as resolved.
}

warm "$HOME/.cache/coursier" ".cache/coursier" coursier
warm "$HOME/.ivy2" ".ivy2" ivy2
15 changes: 4 additions & 11 deletions plugins/scala-sbt/default.nix
Original file line number Diff line number Diff line change
@@ -1,21 +1,10 @@
let
home = builtins.getEnv "HOME";
mountIf = path: target:
if home != "" && builtins.pathExists (home + path)
then [ { source = "~" + path; inherit target; } ]
else [ ];
in
{
nix.packages = [
"sbt"
"scala"
"scala-cli"
];

mounts =
mountIf "/.cache/coursier" "~/.cache/coursier"
++ mountIf "/.ivy2" "~/.ivy2";

network.domains =
[
"maven.org"
Expand All @@ -27,4 +16,8 @@ in
);

scripts = [ ./scripts/setup.sh ];

# Warm coursier/ivy2 caches from host on every up. The command is
# sentinel-guarded inside the guest, so re-runs are no-ops once warmed.
hooks.post-up = [ "nixbox scala-sbt warm-cache" ];
}
Loading