sim: PolicyRunner silently swallows on_frame exceptions — risk of empty datasets

[cagataycali]

Problem

In PolicyRunner (or wherever on_frame is dispatched), exceptions from the
hook are caught and logged at WARN level, then iteration continues:

try:
    on_frame(step_count, observation, action_dict)
except CooperativeStop:
    raise
except Exception as e:
    logger.warning("on_frame hook raised: %s", e)

Failure mode: if the recording hook has a real bug (e.g. obs["missing_key"]
or a dead mount point on the video writer), it throws on every step. A
500-step episode produces 500 log lines, then completes "successfully" with
zero frames written. The dataset is silently empty. The agent learns nothing.

Fix

Count consecutive on_frame failures. After N in a row (5? 10?), raise and
fail the episode with the underlying exception chained.

_consecutive_onframe_failures = 0
_MAX_CONSECUTIVE_FAILURES = 5

try:
    on_frame(step_count, observation, action_dict)
    _consecutive_onframe_failures = 0
except CooperativeStop:
    raise
except Exception as e:
    _consecutive_onframe_failures += 1
    logger.warning("on_frame hook failed (%d/%d): %s",
                   _consecutive_onframe_failures, _MAX_CONSECUTIVE_FAILURES, e)
    if _consecutive_onframe_failures >= _MAX_CONSECUTIVE_FAILURES:
        raise RuntimeError(
            f"on_frame hook failed {_MAX_CONSECUTIVE_FAILURES} times in a row; aborting episode"
        ) from e

Acceptance

• Threshold documented + configurable (env var or PolicyRunner ctor arg)
• Regression test: hook that always raises → episode fails fast instead of
  silently completing
• Regression test: hook that raises once then recovers → counter resets

---

Surfaced by a second-opinion review of PR #85.

[cagataycali]

Landed on PR #85 in commit daaf421.

Fix: PolicyRunner.run now counts consecutive on_frame failures. After N in a row (default 5, overridable via new max_onframe_failures kwarg), raises RuntimeError so run() returns status='error'. Counter resets on the next successful call.

Threaded the kwarg through:

• PolicyRunner.run (core)
• SimEngine.run_policy (base)
• Simulation.run_policy (MuJoCo override)

Tests: 4 new in TestOnFrameFailureCounter:

• test_single_onframe_failure_is_tolerated
• test_consecutive_onframe_failures_abort_episode
• test_consecutive_counter_resets_on_success
• test_default_threshold_is_5

Will close on PR #85 merge.