fix(scheduler): reclaim orphaned thread claims (unblock prod hot-loop, fixes #1559)

[bingran-you]

Summary

• Cherry-picks c1792f8 "Reclaim orphaned scheduler thread claims" onto latest dev as a narrow, conflict-free PR.
• Unblocks the prod scheduler hot-loop documented in [P0] devops scan 2026-04-23 01:10 UTC: prod fully degraded — c1792f8 not deployed, 0 success in 2h; 2 new stuck staging tasks #1559 where the worker defers ~35 due tasks/second with reason=thread busy because the in-memory running_threads HashMap has stale claims with no matching task_executions running row.
• Same diff that is also bundled into Raise ACI delete timeout from 20s to 120s (fixes #1543) #1544, but Raise ACI delete timeout from 20s to 120s (fixes #1543) #1544 is currently CONFLICTING against dev; this PR is a clean-cut subset so the reliability fix can land without waiting on the ACI-delete-timeout review.

Why this is urgent

Prod (commit 214cfd9) has been wedged since 2026-04-23 01:08 UTC. Two concrete user-facing impacts:

• Email reply to shuangshuang329@gmail.com (task b0359ed4) stuck ≈28h.
• Email reply to 1441903771@qq.com (EAP essay, task 35b9d323) stuck ≈4h.
  Plus 33 other deferred-but-never-executed tasks across 15+ users.

What the patch does

scheduler_module/src/service/scheduler.rs:

• Adds try_reclaim_orphaned_thread_claim that, when a claim is refused as "thread busy", checks whether the claim's task has any matching running row in task_executions. If not, the entry is dropped from running_threads and the slot is reclaimed.
• Wires reclaim attempts into claim_thread_execution_slot so the hot-loop self-heals within a tick instead of wedging forever.

Test plan

• cargo build -p scheduler_module --release green
• Once merged, deploy to prod and confirm found N due task(s) followed by executing (not deferred) for most tasks within one tick
• Watch task_executions running count on prod return to near-zero once legitimate ACI runs finish

Follow-ups (not part of this PR)

• Raise ACI delete timeout from 20s to 120s (fixes #1543) #1544 — ACI delete timeout raise (can be rebased after this lands)
• fix(scheduler): prevent hot-loop on cron tasks when workspace thread is busy #1540 — cron hot-loop (distinct root cause)
• fix(scheduler): recover orphaned mongo executions in ~30 min instead of ~20 h #1499 — shorten watchdog

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
dowhiz	Ready	Preview, Comment	Apr 23, 2026 2:23am