Sessions show 'running' state after underlying process exits

[arcavenai]

Summary

When a session's pane process exits (crash, missing binary, etc.), marvel get sessions continues to report state: running. The state never transitions to crashed, exited, or error. The only signal is the health column eventually flipping to unhealthy after the heartbeat timeout window (~30s).

Observed

WORKSPACE  TEAM   ROLE        GEN  NAME                   STATE    HEALTH   CTX%  DESK  AGENT
demo       squad  agent       1    squad-agent-g1-0       running  unknown  -     6     simulator
demo       squad  agent       1    squad-agent-g1-1       running  unknown  -     7     simulator
demo       squad  agent       1    squad-agent-g1-2       running  unknown  -     8     simulator
demo       squad  supervisor  1    squad-supervisor-g1-0  running  unknown  -     9     simulator

All show running / unknown even though bin/simulator doesn't exist and the pane exited immediately.

Expected

The daemon should detect pane exit promptly (poll tmux pane existence or watch for process exit) and transition session state to crashed or exited. Suggested states:

• exited — process terminated with exit code 0
• crashed — process terminated with non-zero exit or signal
• error — command failed to launch (not found, permission denied)

This would give operators immediate visibility without waiting for healthcheck timeout.

Environment

• marvel 0.1.0-alpha.20260417.040512.b72898e, aarch64, Debian 13

[arcaven]

This matches a kinu-side bd issue I filed earlier (aae-orc-8ci) with the same symptom and proposed fix: add pane-existence check to the health reconciler so state flips to 'failed' when the pane vanishes, independent of heartbeat timeouts. Work tracked together — I'll close this when aae-orc-8ci lands.

[arcavenai]

Tested in marvel 0.1.0-alpha.20260418.025039.50c78ae. Partial improvement.

Diagnostic side is much better — pane exit is detected within ~1s and clearly logged:

session util/tools-shell-g1-0: pane %1 gone, reaping
session util/tools-shell-g1-0 exec: sh
session util/tools-shell-g1-0 running in pane %2

Reproduced by marvel work examples/generic-agent.yaml, then marvel inject util/tools-shell-g1-0 "exit" -e.

However, the STATE column in marvel get sessions still always reads running because the daemon respawns immediately — an operator using only the CLI (not tailing the daemon log) never sees a crashed/exited transient. The "Expected" section of this issue specifically asked for state column transitions; that piece isn't visible from the CLI yet.

[arcaven]

Follow-on to the partial-improvement report: the CLI visibility piece is in #24 alongside the #11 reap-path fix. New SessionCrashed state — ReapDead marks the session Crashed with an empty PaneID instead of deleting, so it stays visible via marvel get sessions for the full backoff window. The reconciler clears the Crashed marker at the moment it spawns a replacement. Bounded: at most one Crashed marker per role even in saturation. Ready for re-verification on the next alpha.

[arcavenai]

Verified fixed on alpha-20260419.014522.b8f1f4b (today's build via PR #24 + PR #25).

Test: fresh daemon, crashloop manifest (sh -c 'exit 1', restart_policy: always, process-alive healthcheck), observed via --cluster desk over mrvl://.

$ marvel --cluster desk get sessions
WORKSPACE  TEAM   ROLE       GEN  NAME                  STATE    HEALTH   CTX%  DESK  AGENT
crashloop  flaky  exit-fast  1    flaky-exit-fast-g1-0  crashed  unknown  -           shell

State correctly transitions running → crashed when the process exits (no longer stuck at running). PaneID is cleared on the crashed marker (DESK column blank). Transient crashed state is visible in marvel get sessions for the full backoff window before the reconciler restarts.

Events emitted in parallel (PR #25):

$ marvel --cluster desk events -n 10
TIME      SEV      KIND             SESSION                         MESSAGE
03:13:19  info     session.created  crashloop/flaky-exit-fast-g1-0  pane %1
03:13:19  warning  session.crashed  crashloop/flaky-exit-fast-g1-0  pane %1 gone
03:14:21  info     session.created  crashloop/flaky-exit-fast-g1-0  pane %2
03:14:23  warning  session.crashed  crashloop/flaky-exit-fast-g1-0  pane %2 gone

Closing.