fix(scheduler): tolerate already-finalized rows in record_execution_finish (#1459)

[bingran-you]

Summary

record_execution_finish required status: \"running\" in its Mongo filter. When the stale-execution reconciler or the dowhiz-service-debug audit flips a row to failed/superseded before the worker's finish call lands, matched_count == 0 and the function returns SchedulerError::Storage. That propagates into after_execute_failed and — for OneShot RunTask entries — eventually disables the task, even though the agent may have actually completed its work.

Evidence

Today's production sweep (dowhiz_production_little_bear, dw_worker 118 restarts) surfaced three instances of this cascade — see #1459 audit comment:

```
2026-04-21T17:29:37Z ERROR scheduler task 7161c400-d45d-54a6-26c1-530fb5432815
for user 8574d066-... failed: storage error: missing running execution row ...
2026-04-21T19:02:23Z ERROR scheduler task d04f410f-1e2c-1d35-3208-0f84890327ab
for user 2aedb09a-... failed: storage error: missing running execution row ...
2026-04-21T19:02:51Z ERROR scheduler task eaa96484-8ccc-692c-2060-6ce150904025
for user 8574d066-... failed: storage error: missing running execution row ...
```

Task `7161c400` is now `enabled=false` in prod Mongo (user inbound to `erics_toy@icloud.com`) — the direct user-visible outcome of this cascade.

Change

• DoWhiz_service/scheduler_module/src/scheduler/store/mongo.rs: when record_execution_finish's update matches no row, warn-log and return Ok(()) instead of erroring. The scheduler's intent is already satisfied.
• No change to reconcile_stale_running_executions_for_task's matched_count assertion — that path has different invariants.

Complementary to PR #1499

#1499 reduces how long a row can stay orphaned (20h → 30min). This PR prevents a successful agent run from being mis-reported as a scheduler failure when that reconciler wins the race.

Test plan

• cargo check -p scheduler_module --lib — clean (only pre-existing warnings).
• Deploy to staging; reproduce by marking a running row failed mid-task, confirm the worker logs the warn line and tasks.enabled is not flipped to false.
• Production canary: watch for the missing running execution row error to disappear from dw-worker-error.log.

Refs #1459.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
dowhiz	Ready	Preview, Comment	Apr 21, 2026 8:29pm

[bingran-you]

Follow-up landed in bf2471a5 to narrow the race handling in record_execution_finish:

• if the execution row still exists but is already terminal, we warn and treat the finish as best-effort
• if the execution row is actually missing, we still return SchedulerError::Storage instead of swallowing the mismatch

Added Mongo-backed regressions for both cases and reran:

• MONGODB_URI=mongodb://localhost:27017 MONGODB_DATABASE=codex_pr1516_regression cargo test -p scheduler_module --lib record_execution_finish_ -- --nocapture
• MONGODB_URI=mongodb://localhost:27017 MONGODB_DATABASE=codex_pr1516_scheduler_basic cargo test -p scheduler_module --test scheduler_basic

Both passed locally.

This reply was drafted by breeze, an autonomous agent running on behalf of the account owner.

[bingran-you]

Follow-up landed in bf2471a5 to narrow the race handling in record_execution_finish:

• if the execution row still exists but is already terminal, we warn and treat the finish as best-effort
• if the execution row is actually missing, we still return SchedulerError::Storage instead of swallowing the mismatch

Added Mongo-backed regressions for both cases and reran:

• MONGODB_URI=mongodb://localhost:27017 MONGODB_DATABASE=codex_pr1516_regression cargo test -p scheduler_module --lib record_execution_finish_ -- --nocapture
• MONGODB_URI=mongodb://localhost:27017 MONGODB_DATABASE=codex_pr1516_scheduler_basic cargo test -p scheduler_module --test scheduler_basic

Both passed locally.

This reply was drafted by breeze, an autonomous agent running on behalf of the account owner.