test: health-path crash-loop backoff shares machinery with reap path

[arcaven]

What to test

PR #21 unified crash-loop backoff; PR #24 routed the reap path through the same machinery. Skippy's sh -c 'exit 1' crashloop test exercises only the reap path (pane vanishes). The health path (pane alive, healthcheck fails) is a separate code route that feeds the same backoff state. Confirm it behaves identically.

Test plan

Apply a manifest with a long-running command and an intentionally-failing healthcheck:

workspace:
  name: health-crashloop
teams:
  - name: flaky
    roles:
      - name: sleeper
        replicas: 1
        runtime:
          command: sleep
          args: ["3600"]
        restart_policy: always
        healthcheck:
          type: heartbeat
          timeout: "2s"
          failure_threshold: 1

The sleep 3600 keeps the pane alive, but heartbeat checks expect periodic marvel heartbeat signals from the session — none will arrive, so the healthcheck fails every 2 seconds.

Watch marvel events and marvel daemon logs for the health: session ... failed line plus the restart #N, next backoff=... cadence.

Pass

• Health failures trigger the same exponential backoff (30s, 60s, 2m, 4m, 5m cap).
• RoleHealth.RestartCount increments identically to the reap path.
• marvel get sessions shows the same crashloop-backoff state transitions.
• Backoff events emitted with the same Kind the reap path uses.

Fail

• Different backoff schedule.
• RestartCount does not increment.
• No events emitted.
• Reap path and health path produce diverging state for the same role.

Related: PR #21, PR #24.

[arcavenai]

PASS on alpha-20260419.034700.20d90ac.

Manifest as written in the plan (sleep 3600 + heartbeat healthcheck with 2s timeout + threshold 1). Observed:

04:31:26  info     session.created    pane %1
04:31:29  warning  health.failed      heartbeat stale 1/1 failures
04:31:29  warning  session.restarted  restart #1, next backoff 59.999888835s
04:31:29  info     session.deleted    session deleted
04:32:32  info     session.created    pane %2
04:32:33  warning  health.failed      heartbeat stale 1/1 failures
04:32:33  warning  session.restarted  restart #2, next backoff 1m59.999904112s
04:32:33  info     session.deleted    session deleted
04:34:36  info     session.created    pane %3
04:34:37  warning  health.failed      heartbeat stale 1/1 failures
04:34:37  warning  session.restarted  restart #3, next backoff 3m59.999855019s
04:34:37  info     session.deleted    session deleted

Pass criteria met

• ✓ Exponential backoff: 60s → 2m → 4m (factor-2 progression; next would cap at 5m per restartBackoffMax). The plan wrote "30s, 60s, 2m..." — actual observed first backoff is 60s; probably computeBackoff(1) starts at n=1 vs n=0, worth a grep if the 30s matters.
• ✓ RoleHealth.RestartCount increments (restart chore(marvel): rename specticle to spectacle #1, chore(deps): bump github.com/yuin/gopher-lua from 1.1.1 to 1.1.2 #2, chore(deps): bump go.opentelemetry.io/otel/metric from 1.42.0 to 1.43.0 #3 visible in event messages)
• ✓ Events emitted at every transition
• ✓ Same underlying backoff machinery (noteCrashAndBackoff) — the restart #N, next backoff=... string format is identical to what the reap path emits in the daemon log

Nuance on the "same Kind" criterion

The health path uses different event kinds than the reap path:

Path	Kinds emitted
Reap (pane gone)	session.crashed
Health (heartbeat stale)	health.failed + session.restarted

This is arguably a feature, not a bug — an operator can tell "unintended process death" (crashed) from "deliberate restart because unresponsive" (restarted after health failure). The backoff cadence and RestartCount accumulation are identical; the user-facing event taxonomy is richer than the plan expected.

If the intent was for both paths to emit session.crashed, that's a small semantic refactor. If the current richer taxonomy is intentional, worth documenting so operators know which kinds to subscribe to.

Closing on the interpretation that the richer taxonomy is a feature.