test: marvel shift rolling-replace a team's sessions

[arcaven]

What to test

marvel shift <team> performs a rolling replacement: drain old-generation sessions one at a time while launching new-generation replacements. Never exercised on desk.

Test plan

1. Apply a healthy team:

workspace:
  name: shift-test
teams:
  - name: workers
    roles:
      - name: w
        replicas: 3
        runtime:
          command: sleep
          args: ["3600"]

2. Wait until marvel get sessions shows 3× running sessions with gen=1.
3. marvel shift shift-test/workers.
4. marvel get sessions -w or tail marvel events.
5. Expect: one-at-a-time replacement — old gen 1 session deleted, new gen 2 session created, cycle through all 3.

Pass

• 3 drain + 3 launch events, interleaved per the reconcile cadence.
• Final state: 3× gen=2 running, 0× gen=1.
• No observed window where actual < desired for the role.
• One shift event per session transition (shifted/drained/launched — whichever kinds are emitted).
• marvel describe team shift-test/workers shows Shift.Phase = ShiftNone at the end.

Fail

• All sessions dropped at once before replacements come up.
• Shift stalls with mixed generations (reconciler gets stuck).
• More than 3 replicas alive simultaneously.
• Generation never advances past 1.

Edge

• Apply a heterogeneous team (two roles). Shift one role only. Other role's sessions should be untouched and stay on gen=1.

Related: internal/team reconcileShift.

[arcavenai]

Mixed result on alpha-20260419.034700.20d90ac.

Main test (single-role team, 3 replicas)

Gen advanced 1→2, all old sessions replaced, team.shift-started and team.shift-completed events emitted, Shift.Phase returned to empty.

But — one fail criterion hit

Plan says: "Fail: More than 3 replicas alive simultaneously."

Observed timeline:

04:29:39  session.created  g1-0 g1-1 g1-2       ← apply, 3 replicas
04:29:44  team.shift-started  gen 1→2 roles=[w]
04:29:44  session.created  g2-0 g2-1 g2-2       ← all 3 new replicas spawn at once
# 04:29:44 → 04:29:46 — 6 sessions alive (3× gen1 + 3× gen2)
04:29:46  session.deleted  g1-0
04:29:47  session.deleted  g1-2
04:29:49  session.deleted  g1-1
04:29:53  team.shift-completed  gen 2 active

For ~2 seconds there were 6 replicas alive when desired is 3. That's blue-green, not rolling. Consequences:

• Double the tmux pane count (and any per-session resources) transiently
• If role sessions share scarce state, twice the pressure during shift
• The "never actual < desired" invariant holds fine (plan's other bullet), but the inverse — "never actual > desired by more than maxSurge" — isn't enforced

A proper rolling strategy (maxUnavailable or maxSurge kubernetes-style) would either drain one old before launching one new, or cap surge to e.g. +1. Here the implementation is launch-all-new-first-drain-all-old-after.

Edge case — heterogeneous team, --role scoping: PASS

teams:
  - name: mixed
    roles:
      - name: a
        replicas: 2
      - name: b
        replicas: 2

marvel shift hetero-shift/mixed --role a → role a advanced to gen=2, role b stayed at gen=1. Clean.

mixed  a  2  mixed-a-g2-0  running
mixed  a  2  mixed-a-g2-1  running
mixed  b  1  mixed-b-g1-0  running    ← untouched
mixed  b  1  mixed-b-g1-1  running    ← untouched

Suggestion

Either explicitly document the blue-green behavior as the current semantic (and revise the pass criteria here), or introduce a surge/max-unavailable control. Most operator expectations around the word "shift" (and kubectl's rollout) will assume rolling unless documented otherwise.

Leaving open for the surge behavior. Role-scoping edge case is confirmed working.