rc=143 failures on staging caused by earlyoom killing Claude processes under memory pressure

[nathanschram]

Summary

Two related issues causing Claude Code failed (rc=143) on staging:

1. /new command doesn't cancel running processes (fixed)

/new only cleared stored resume tokens — it did NOT cancel running processes. If a Claude session was running and the user typed /new, the old process kept running (~400 MB RSS), and the next prompt spawned a new session. This leaked processes and worsened memory pressure.

Fix: /new now cancels all running tasks for the chat before clearing sessions. Reply message shows "cancelled run and cleared" when a process was killed.

2. earlyoom killing Claude processes under memory pressure

Root cause of the rc=143: earlyoom sends SIGTERM to Claude Code processes when system memory drops below thresholds. Not an Untether or Claude Code bug — earlyoom is correctly protecting the system.

Evidence (earlyoom)

earlyoom logs (2026-03-23) — every rc=143 maps to an earlyoom kill:

Time	earlyoom action	Untether session
15:54:08	SIGTERM → PID 1716517 "claude" (424 MB)	lba-web rc=143
16:00:02	SIGTERM → PID 1832869 "claude" (394 MB)	scout rc=143
16:02:35	SIGTERM → PID 1850322 "claude" (384 MB)	cf-monitor rc=143
16:02:50	SIGTERM → PID 1868362 "claude" (374 MB)	lba-web rc=143
16:05:14	SIGTERM → PID 1944950 "claude" (379 MB)	scout rc=143
16:05:49	SIGTERM → PID 1947761 "claude" (366 MB)	cf-monitor rc=143
16:11:39	SIGTERM → PID 1977747 "claude" (379 MB)	scout rc=143
16:11:54	SIGTERM → PID 1974757 "claude" (371 MB)	3c317ef4 lba-web rc=143

Root cause: memory saturation

• 31 GB total RAM, swap at 0% free
• 128 workerd processes from scout consuming ~13.5 GB
• 3-4 concurrent Claude sessions at ~400 MB each
• earlyoom triggers at <10% free RAM AND <5% free swap

Changes

• src/untether/telegram/commands/topics.py — Added _cancel_chat_tasks() helper; updated _handle_new_command and _handle_chat_new_command to cancel running tasks before clearing sessions
• src/untether/telegram/loop.py — Added running_tasks to TelegramCommandContext; passed through all /new call sites
• tests/test_telegram_topics_command.py — 10 new tests covering cancellation behaviour

Affected versions

• Untether: v0.35.0rc11
• Claude Code: v2.1.81
• Host: lba-1 (Hetzner, 30 GB RAM, 4 GB swap)

[nathanschram]

Update: Root cause is orphaned grandchild processes, not just memory pressure

Finding

The 325 workerd processes (27 GB) are orphaned grandchildren of Claude sessions run via Untether:

1. Claude (via Untether Bash tool) runs npx vitest run in the scout project
2. vitest spawns ~50-100 workerd processes
3. earlyoom kills vitest/claude mid-test (memory spike)
4. workerd processes orphan to systemd (PPID=1754) but stay in untether.service cgroup
5. Next session runs vitest again → spawns MORE workerd
6. Cascading failure: 325 orphans × ~84 MB = 27 GB

Evidence:

• All 325 workerd have PPID=1754 (user systemd) — orphaned
• All created at 19:09 — exactly when the failed scout session ran vitest
• All in untether.service cgroup: /user.slice/user-1000.slice/user@1000.service/app.slice/untether.service
• No wrangler/miniflare parent processes alive

Why only via Untether?

Untether uses start_new_session=True for Claude subprocesses. When earlyoom kills Claude mid-tool, grandchild processes (workerd, node) re-parent to systemd instead of being cleaned up. In tmux, the terminal session's process group management handles cleanup differently.

Actions needed

1. /new should kill grandchild processes (not just the direct subprocess)
2. Session cleanup on completion/cancellation should kill the entire process tree
3. /restart drain should also clean up orphaned process trees
4. Consider cgroup-level cleanup on session end

[nathanschram]

Update 2: Full root cause — orphans in cgroup, NOT Claude Code memory leak

The 6 Claude processes are NOT from Untether

All 6 running Claude processes are in tmux scopes (manual CLI sessions), not the untether.service cgroup:

PID=982418  CG=tmux-spawn-18c92846...  RSS=557MB
PID=1356305 CG=tmux-spawn-3fa6cb11...  RSS=557MB
PID=1524922 CG=tmux-spawn-a09bfce2...  RSS=454MB
...

The untether.service cgroup contents

untether (python)       — 1 process
trello MCP              — 2 processes  
telegram MCP            — 1 process
orphaned workerd        — 325 processes (15.4 GB!)

The 325 workerd processes were spawned by Claude's Bash tool running npx vitest run in the scout project. When earlyoom killed the session mid-test, workerd orphaned to systemd but remained in the untether.service cgroup. Each subsequent test run added more orphans.

Why only via Untether?

1. tmux: Each session runs in its own tmux-spawn-*.scope cgroup. Orphaned grandchildren stay in that scope. earlyoom sees them as separate from other sessions.

2. Untether: All sessions share one untether.service cgroup. Orphaned grandchildren accumulate in that cgroup, inflating its apparent memory footprint. earlyoom targets the process with the highest oom_score, which is influenced by cgroup membership.

3. The cascade: earlyoom kills Claude mid-vitest → workerd orphans → cgroup grows → next session triggers earlyoom sooner → more orphans → repeat.

Cleanup result

pkill -f 'workerd serve --binary'
Before: 25 GB used, untether cgroup 15.4 GB
After:  11 GB used, untether cgroup 645 MB

Structural fix needed

On session end (completion, cancellation, or earlyoom kill), Untether should clean up orphaned processes in the cgroup that were spawned during that session. Options:

• Track child PIDs at spawn time, kill descendants at session end
• Use systemd-run --scope per Claude session for automatic cgroup isolation
• Add periodic orphan detection in the watchdog

[nathanschram]

Update 3: Root cause confirmed — auto-continue death spiral

The death spiral

When earlyoom kills Claude sessions, session.auto_continue triggers because all killed sessions have last_event_type=user (Claude received a tool result but was killed before producing a response). This matches the auto-continue detection pattern designed for upstream Claude Code bugs #34142/#30333.

But auto-continue can't distinguish "upstream bug" from "earlyoom killed the process."

Timeline from 19:09-19:10 today:

19:09:46  lba-web       rc=-9   (earlyoom SIGKILL)
19:09:49  brand-copilot rc=143  (earlyoom SIGTERM)
19:09:49  → auto-continue lba-web (last_event_type=user)
19:09:52  → auto-continue brand-copilot
19:09:53  untether      rc=-9   (earlyoom SIGKILL)
19:09:53  → auto-continue untether
19:09:55  scout         rc=143  (the user's error: 0334d88d)
19:09:57  → auto-continue scout
19:10:02  lba-web       rc=143  (spawned 7s ago, 0 events, instantly killed)
19:10:03  untether      rc=143  (spawned 7s ago, 0 events, instantly killed)
19:10:05  brand-copilot rc=143  (spawned 7s ago, 0 events, instantly killed)
19:10:15  scout         rc=143  (spawned 12s ago, 0 events, instantly killed)

Each auto-continue spawns a new Claude process (~400 MB) + MCP servers (~800 MB) = ~1.2 GB. Four concurrent auto-continues = ~5 GB of new memory instantly on an already-saturated system. They all die within seconds.

Why it ONLY happens via Untether (not CLI)

Auto-continue doesn't exist in CLI. In CLI:

1. earlyoom kills Claude → it dies
2. Memory pressure drops
3. Other sessions continue fine

In Untether:

1. earlyoom kills 4 Claude sessions
2. Auto-continue immediately re-spawns all 4 (adding ~5 GB)
3. earlyoom kills them all again (0 events, 6-12 second lifespan)
4. max_retries=1 exhausted → finally gives up
5. Meanwhile, orphaned grandchildren (workerd) accumulate

Fix

Auto-continue should NOT trigger on signal deaths (rc=143/SIGTERM, rc=-9/SIGKILL, rc=137/SIGKILL). These are external kills, not the upstream bug. The detection should check proc.returncode — the upstream bug exits with rc=0, not a signal code.

Additionally, auto-continue should be suppressed when the session had 0 events (duration <15s) — this indicates the process never even started successfully.

[nathanschram]

Update 4: Two contributing factors, not just one

Factor 1: earlyoom + auto-continue death spiral (confirmed)

As documented in Update 3. Auto-continue should not trigger on signal deaths.

Factor 2: Possible upstream Claude Code bug — headless SIGTERM deaths

anthropics/claude-code#29642 reports that headless Claude Code sessions (--output-format stream-json) die with SIGTERM (rc=143) after 3-10 minutes of active work, even with only one session running and no memory pressure. This is the EXACT invocation Untether uses.

Key observations from the upstream issue:

• Affects long-running headless sessions (5-10+ minutes)
• Process receives SIGTERM mid-work with no error in output stream
• Tasks with WebSearch/WebFetch die faster
• Sessions that go quiet (no output for 2+ min) more susceptible
• Hypothesis: shared AbortController + OAuth token refresh timing
• No fix from Anthropic yet (issue is stale label)

This would explain why the user sees rc=143 only via Untether (which uses --output-format stream-json) and never in CLI (which uses the interactive TUI mode). The headless stream-json mode may have a different lifecycle/timeout behaviour.

Screenshots analysis

The screenshots show two patterns:

1. rc=-9 (SIGKILL) on lba-web at 6:56 PM — earlyoom escalated
2. rc=143 (SIGTERM) on brand-copilot at 6:36 PM, scout at 7:04 PM — could be earlyoom OR the upstream headless bug

The auto-continue messages appear in the Untether Staging notification chat, while errors appear in project-specific chats — so they can appear in either order depending on Telegram delivery timing. They're from the same event.

What rc=143 means

rc=143 = 128 + 15 = SIGTERM (signal 15). The process received a polite termination request. Sources:

• earlyoom sending SIGTERM under memory pressure
• The upstream headless bug (Claude Code self-terminates)
• Any external kill <pid> or kill <pgid>

Fix plan

1. Auto-continue: Don't trigger on signal deaths (rc=143, rc=-9/137). The upstream bug #34142/#30333 exits with rc=0.
2. Investigate: Whether rc=143 deaths still occur after clearing orphaned workerd and with no memory pressure — if so, it's the upstream headless bug.
3. Upstream: Monitor [BUG] Headless single-agent sessions die with SIGTERM after 3-10 minutes (Max subscription, no rate limits) anthropics/claude-code#29642 for a fix.

[nathanschram]

Update 5: Upstream headless bug RULED OUT — pure earlyoom + auto-continue

Evidence: all rc=143 deaths are clustered mass kills

Every single rc=143 in the last 3 days occurs as part of a simultaneous multi-project kill event. There is NOT a single isolated rc=143:

Kill event	Time window	Deaths	Projects affected
Mar 22 19:01-19:12	11 min	6	scout, wpnav, lba, platform, brand
Mar 23 15:54-16:11	17 min	8	lba, scout, cf
Mar 23 19:09-19:10	1 min	8	lba, brand, untether, scout (+ auto-continue respawns)

If the upstream headless bug (#29642) were involved, we'd see isolated deaths at random times. Instead, ALL projects die simultaneously — classic earlyoom behaviour.

Longest successful sessions prove no headless timeout

Duration	Events	Project
54 min	876	scout
49 min	6	brand-copilot
37 min	1022	platform
29 min	788	cf-monitor
27 min	788	brand-copilot
26 min	128	scout
26 min	391	untether

Sessions routinely run 25-54 minutes via --output-format stream-json with no issues. The upstream bug #29642 kills at 3-10 minutes — that's not happening here.

wpnav specifically

10 successful sessions (rc=0), only 2 rc=143 — both at 19:01 and 19:12 on Mar 22, during the earlyoom cascade that killed ALL projects. Every wpnav session outside kill windows completed normally.

Conclusion

The root cause is earlyoom + auto-continue death spiral, not a Claude Code headless timeout. Fix: suppress auto-continue on signal deaths.

[nathanschram]

Progress: auto-continue fix implemented

PR #226 suppresses auto-continue on signal deaths:

• _is_signal_death() checks rc>128 (shell convention) and rc<0 (Python convention)
• _should_auto_continue() now passes proc_returncode through from JsonlStreamState
• 12 new tests covering all signal/normal exit code combinations
• Full suite: 1815 tests pass, 81.37% coverage

This prevents the death spiral where earlyoom kills → auto-continue respawns 4 sessions → earlyoom kills again.

Combined with PR #223 (/new cancels running tasks), both contributing factors are addressed:

1. fix: /new command now cancels running processes (#222) #223 (merged): /new kills old processes instead of leaking them
2. fix: suppress auto-continue on signal deaths (#222) #226 (pending): auto-continue doesn't respawn signal-killed sessions