Zombie process accumulation when openab runs as PID 1

[agent-worker-use]

Description

When openab runs as PID 1 in a container, it does not reap orphaned child processes, leading to zombie process accumulation over time.

• openab runs as PID 1 (container entrypoint: openab run /etc/openab/config.toml)
• openab forks kiro-cli (ACP mode), which in turn forks worker processes (e.g., python3, pkill)
• When these worker processes terminate and their direct parent has already exited, they are reparented to PID 1 (openab)
• openab does not call wait() or waitpid(), causing these orphaned processes to remain as zombies (Z state)
• After 2 days of container runtime, 9 zombie processes accumulated, all with PPID=1
• Zombies consume no CPU or memory, but occupy PID table entries; long-running containers risk PID space exhaustion

Steps to Reproduce

1. Start openab as container PID 1 (openab run config.toml)
2. Trigger kiro-cli via ACP to execute shell subprocesses (e.g., subprocess.run() launching python3)
3. Allow worker processes to complete normally
4. Observe with ps aux — zombie processes appear with Z (defunct) state and PPID=1

Expected Behavior

Zombie processes should be reaped and removed from the process table. Suggested solutions:

1. SIGCHLD Handler: Register a SIGCHLD signal handler that calls waitpid(-1, WNOHANG) to reap zombies
2. Child Subreaper: Use prctl(PR_SET_CHILD_SUBREAPER) combined with reaper logic
3. Init Wrapper: Document the recommendation to use a dedicated init process (e.g., tini, dumb-init) as the container entrypoint

Environment

• openab version: 0.7.3
• Container runtime: Kubernetes pod (k3s)
• OS: Linux

Screenshots / Logs

$ ps -eo pid,ppid,stat,comm | grep Z
65 1 Z pkill
67 1 Z python3
81 1 Z python3
87 1 Z python3
158 1 Z python3
436 1 Z python3
462 1 Z python3
533 1 Z python3
735 1 Z pkill

This is a known issue with PID 1 processes in containers. The Go runtime does not automatically handle SIGCHLD, so explicit zombie reaping is required when running as PID 1.

[thepagent]

Moving to Massami for reproduction.

[thepagent]

related to #290

[masami-agent]

Reproduce Results

Environment: openab 0.7.7, Helm chart 0.7.7, test pod running on K8s (openab as PID 1, no init wrapper)

Steps

1. Deployed a fresh test pod with openab 0.7.7 as PID 1 (standard image, no tini)
2. Triggered tool calls via Discord that execute shell subprocesses (sleep 60, sleep 120)
3. Observed that kiro-cli-chat spawns subprocesses through a shell wrapper; the wrapper exits first, and the actual subprocess gets reparented to PID 1 (openab)
4. After the subprocess finishes, it remains as a zombie — openab does not call waitpid()

Result

pid=1      ppid=0      state=S  openab         ← PID 1, no SIGCHLD handler
pid=35     ppid=1      state=S  kiro-cli
pid=42     ppid=35     state=S  kiro-cli-chat
pid=322    ppid=1      state=Z  sleep          ← zombie (from sleep 60 tool call)
pid=368    ppid=1      state=Z  sleep          ← zombie (from sleep 120 tool call)

Both zombies have ppid=1 and persist indefinitely. Confirmed: openab does not reap orphaned grandchildren when running as PID 1.

Root Cause

In src/acp/connection.rs, openab spawns the agent CLI as a direct child with kill_on_drop and process group management. However, when the agent (kiro-cli-chat) executes tool calls, it spawns subprocesses through intermediate shell wrappers. If the wrapper exits before the actual command, the command gets reparented to PID 1 (openab). When that command finishes, it becomes a zombie because main.rs has no SIGCHLD handler and never calls waitpid(-1, WNOHANG).

This is the same root cause as #290.

---

Triage Recommendation

• Type: bug (confirmed)
• Priority: p2 — zombies consume no CPU/memory but occupy PID table entries. With default pid_max=32768, long-running pods will eventually hit PID exhaustion (~1000 hours at ~30 zombies/hour per bug: openab as PID 1 leaves zombie processes from agent subprocesses #290 data)
• Duplicate of: bug: openab as PID 1 leaves zombie processes from agent subprocesses #290 — same root cause, different reporter and agent variant (kiro-cli vs codex-acp). Recommend tracking the fix in bug: openab as PID 1 leaves zombie processes from agent subprocesses #290.

---

Suggested Fix

Option A: Add tini to all Dockerfiles (recommended, minimal change)

All four Dockerfiles (Dockerfile, Dockerfile.claude, Dockerfile.codex, Dockerfile.gemini) currently run openab directly as PID 1:

ENTRYPOINT ["openab"]
CMD ["run", "/etc/openab/config.toml"]

Fix — add tini as the init process:

RUN apt-get update && apt-get install -y --no-install-recommends tini && apt-get clean && rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["tini", "--"]
CMD ["openab", "run", "/etc/openab/config.toml"]

This is the standard container best practice. tini handles SIGCHLD reaping and signal forwarding automatically. Fixes all agent variants with a 2-line change per Dockerfile.

Option B: Add SIGCHLD reaper in openab (defense-in-depth)

Add a zombie reaper task in main.rs for environments where tini is not available:

// In main.rs, after spawning the cleanup task:
if std::process::id() == 1 {
    tokio::spawn(async {
        use tokio::signal::unix::{signal, SignalKind};
        let mut sigchld = signal(SignalKind::child()).expect("failed to register SIGCHLD");
        loop {
            sigchld.recv().await;
            unsafe {
                while libc::waitpid(-1, std::ptr::null_mut(), libc::WNOHANG) > 0 {}
            }
        }
    });
}

Recommendation

Do both A and B. Option A is the immediate fix (covers all images). Option B is defense-in-depth for users who build custom images without tini.

[obrutjack]

Duplicate of #290 — tracking fix there.