Reconciler does not respawn sessions after workspace delete + re-apply

[arcavenai]

Summary

After marvel delete workspace <name> followed by marvel work <same-manifest>, the team is re-created but no sessions ever spawn. No events are emitted in the 60+ second window observed.

Reproduce (deterministic on this cluster)

1. Start fresh daemon: marvel daemon --mrvl
2. Apply crashloop manifest (any role that exits quickly works):

   workspace:
     name: crashloop
   teams:
     - name: flaky
       roles:
         - name: exit-fast
           replicas: 1
           runtime:
             image: shell
             command: sh
             args: ["-c", "exit 1"]
           restart_policy: always
           healthcheck:
             type: process-alive
             timeout: "2s"
             failure_threshold: 1

3. Observe one session.created + one session.crashed event. Session sticks in crashed state (separate question whether restart_policy=always should retry in this case, but that's not the primary bug).
4. marvel delete workspace crashloop — observe session.deleted event, workspace cleared.
5. marvel work <same-manifest> — workspace/team is reported ready.
6. Wait 60 seconds.

Expected

A fresh session.created event, new session in get sessions output.

Actual

• marvel get teams shows the team (replicas: 1).
• marvel describe team crashloop/flaky shows Generation 1, CreatedAt correctly updated, Role config correct.
• marvel get sessions returns empty.
• marvel events shows zero new events since the apply. Only the pre-delete events remain visible.

Suspected cause

RoleHealth/backoff state keyed by workspace/team/role may survive workspace delete+recreate, leaving BackoffUntil set far in the future (especially if the previous generation hit saturation in noteCrashAndBackoff). The reconciler's gate then refuses to spawn. But this is a guess — the symptom is "team exists, zero replicas actual, no events, no progress."

Alternatively the Session manager or reconcile queue is stalling after cascade delete (#15) — check whether the cascade cleared everything it needed to.

Environment

• marvel 0.1.0-alpha.20260419.014522.b8f1f4b (commit b8f1f4b)
• Fresh daemon, Linux aarch64 Pi, tmux 3.5a
• mrvl:// (reproduces via Unix socket too)

Impact

Blocks testing of PR #21 / PR #24 crash-loop backoff telemetry — can't accumulate health.crashloop-backoff events if each workspace can only be applied once.

Workaround

Restart daemon between manifest applies.

[arcavenai]

Second reproduction path — via scale.

Setup: running crashloop team (sh -c 'exit 1', exponential backoff in effect, session in crashed state with PaneID cleared).

$ marvel scale crashloop/flaky --role exit-fast --replicas 0
team/crashloop/flaky role/exit-fast scaled to 0 replicas

$ marvel get teams
WORKSPACE  TEAM   ROLE       REPLICAS  RUNTIME
crashloop  flaky  exit-fast  0         shell

$ marvel get sessions
WORKSPACE  TEAM   ROLE       GEN  NAME                  STATE    HEALTH   CTX%  DESK  AGENT
crashloop  flaky  exit-fast  1    flaky-exit-fast-g1-0  crashed  unknown  -           shell

Note: scale set Replicas to 0 but the crashed session record was NOT removed. Expected: scale-to-zero should either reap the crashed record or at minimum not count it against desired replicas.

$ marvel scale crashloop/flaky --role exit-fast --replicas 1
team/crashloop/flaky role/exit-fast scaled to 1 replicas

# 60+ seconds later...
$ marvel get sessions
WORKSPACE  TEAM   ROLE       GEN  NAME                  STATE    HEALTH   CTX%  DESK  AGENT
crashloop  flaky  exit-fast  1    flaky-exit-fast-g1-0  crashed  unknown  -           shell

$ marvel events -n 5   # no new events since before the scale

Same stall: desired=1, actual-running=0, but reconciler never spawns. Likely because the crashed record is still counted as "one replica exists" even though its PaneID is cleared and it has no live process. The reconciler wants "1 replica"; it sees the crashed record and thinks the role is satisfied.

Suggested fix direction: replica counting (in team.Controller.reconcileRole or wherever desired-vs-actual is computed) should only count sessions in running state toward the target, not crashed ones. Or the reap path should drop the crashed record entirely once backoff elapses, rather than keep it as a marker.

The original "workspace delete + re-apply" repro probably has the same underlying cause: the old Session record carrying over after cascade delete, blocking the new generation from spawning.

[arcavenai]

Log directory listing (dev requested) and partial correction of my earlier diagnosis:

$ ls -la ~/.marvel/log/
total 116
drwx------ 2 skippy skippy   4096 Apr 18 22:13 .
drwx------ 5 skippy skippy   4096 Apr 18 22:07 ..
-rw------- 1 skippy skippy 106245 Apr 18 22:27 daemon.log

$ ls -la ~/.marvel/run/
total 12
drwx------ 2 skippy skippy 4096 Apr 18 22:13 .
drwx------ 5 skippy skippy 4096 Apr 18 22:07 ..
-rw-r--r-- 1 skippy skippy    7 Apr 18 22:13 daemon.pid

No gzipped archives — 106 KB / 1 MiB rotation cap. Only .log file present.

Daemon log message frequency (summary):

908  ssh: client connected: michael.pursifull (SHA256:yrV5M1roJXkgB9qeKBQ/5PoNPd1alhvliNjJr4R9G/s)
  5  session crashloop/flaky-exit-fast-g1-0 using generic adapter
  5  session crashloop/flaky-exit-fast-g1-0 exec: sh -c 'exit 1'
  5  session crashloop/flaky-exit-fast-g1-0 running in pane %N
  5  session crashloop/flaky-exit-fast-g1-0: pane %N gone, marking crashed
  1  reap: ... crashed (role crashloop/flaky/exit-fast restart #5, next backoff=4m59.999997574s)

Partial correction to the scale-repro in my previous comment: the reconciler DID respawn as pane %5 at 03:25:33 — five minutes after pane %4 crashed, matching restartBackoffMax = 5 * time.Minute. My 60-second observation window was too short. So the scale 0→1 case is not a stall; it's correctly waiting out the previously-accumulated backoff.

What's still on the table for #29: the original workspace-delete+re-apply repro. That case should reset RoleHealth state because the team is re-created under a new generation (CreatedAt updates). But backoff state keyed by workspace/team/role name may carry over even after workspace delete since name is reused. Will re-test with explicit wait-5-minutes before drawing new conclusions.

Suggestion: either workspace delete should clear RoleHealth for the workspace's teams, OR names should be scoped with generation (workspace/team/role@gen) so re-applied teams start fresh.

[arcaven]

Confirmed the workspace-delete+re-apply hypothesis by reading the code. The stall is caused by team.Controller.roleHealth not being cleared on delete.

Where it's keyed: internal/team/controller.go:29-34 — map[string]*RoleHealth keyed by workspace/team/role (string). State includes RestartCount and BackoffUntil.

What the cascade clears: internal/daemon/daemon.go:580-599 clears sessions, teams, workspace record, and calls sessMgr.CleanupWorkspace(ws.Name). Nothing touches roleHealth.

Why it stalls: re-applying with the same name produces identical map keys. If the prior generation hit MaxRestarts saturation, BackoffUntil = saturationFreezeUntil (far future) — the reconciler gate at controller.go:230 returns early forever. Even without saturation, the accumulated RestartCount means the fresh generation waits wherever the old one left off (restart #5 ≈ 5 min before first spawn, which matches the delay in your scale-case correction).

Fix direction (simplest): team.Controller.ClearRoleHealthForWorkspace(name) and ClearRoleHealthForTeam(ws, team), called from handleDelete's workspace and team branches. No schema change, localized to the cascade path.

Alternative would be generation-scoped keys (workspace/team/role@gen) — cleaner long-term but requires threading generation into every lookup. Not worth it unless something else needs it too.

---

Separately, on your log-frequency breakdown in comment 2 — 908 ssh: client connected vs 20 real events is the exact aae-orc-1d2 scenario. PR #26's logbuf dedup doesn't actually compress this in practice: Go's log.Printf 1-second timestamp prefix is part of the line logbuf receives, so two identical connects seconds apart never match. Filed as bd aae-orc-407l. Fix is to dedup on a content fingerprint that strips the leading YYYY/MM/DD HH:MM:SS prefix.

And re your earlier ask for IP in the ssh log line: PR #31 is up.

[arcavenai]

Thanks — code reading nails it. team.Controller.roleHealth keyed by workspace/team/role with no teardown on cascade delete. Reusing the name on re-apply picks up the stale BackoffUntil / RestartCount.

Your proposed ClearRoleHealthFor{Workspace,Team} on handleDelete is clean. Generation-scoped keys would be nice if anything else needed it, but nothing I can see does.

I'll leave this open for your PR; no further testing needed from here. Closing when the fix merges.

[arcaven]

PR #33 up for the RoleHealth-on-cascade-delete fix. The scale 0→1 case you corrected yourself is orthogonal — if we also want scale-to-1 to reset the counter that's a separate design call; flagged in the PR description.

[arcavenai]

Verified fixed on alpha-20260419.034700.20d90ac (PR #33).

Exact repro sequence, post-fix:

$ marvel work /tmp/crashloop3.yaml      # 03:56:54
workspace/crashloop ready

$ marvel events -n 3
03:56:56  info     session.created  crashloop/flaky-exit-fast-g1-0  pane %1
03:56:57  warning  session.crashed  crashloop/flaky-exit-fast-g1-0  pane %1 gone

$ marvel delete workspace crashloop    # 03:56:58
workspace/crashloop deleted

$ marvel work /tmp/crashloop3.yaml      # 03:57:00
workspace/crashloop ready

# 2 seconds later — no more 5-minute stall:
$ marvel events -n 5 | tail -3
03:56:59  info     session.deleted  crashloop/flaky-exit-fast-g1-0  session deleted
03:57:02  info     session.created  crashloop/flaky-exit-fast-g1-0  pane %1  ← fresh!
03:57:03  warning  session.crashed  crashloop/flaky-exit-fast-g1-0  pane %1 gone

Re-applied generation spawns in seconds, not minutes. ClearRoleHealthForWorkspace + ClearRoleHealthForTeam are clearing accumulated BackoffUntil / RestartCount on cascade delete as designed.

Thanks @arcaven for the quick turn. Closing.