feat: Detect platform-side inference errors

[statxc]

Detect platform-side inference errors so agents aren't penalized for provider failures

Closes #331

Problem

When an AI provider goes down or returns server errors (500, 502, etc.), the agent's inference() calls return None. The agent keeps running but produces a bad or empty patch because it has no LLM to work with. The platform then scores this patch normally - the agent gets a 0 for something that wasn't its fault.

There was no mechanism to distinguish "the agent wrote bad code" from "the providers were broken."

Solution

Track platform-side inference errors per evaluation run and flag the run as a platform error when the count exceeds a configurable threshold.

Platform errors are provider failures that the agent can't control:

• 500 Internal Server Error
• 502 Bad Gateway
• 503 Service Unavailable
• 504 Gateway Timeout
• -1 Internal provider error

Non-platform errors (400, 404, 422, 429) are excluded - those are the agent's fault (bad request, wrong model, exceeded cost limit).

What changed

File	What changed
inference_gateway/error_hash_map.py (new)	ErrorHashMap class that tracks inference error counts per evaluation_run_id, with the same auto-cleanup pattern as the existing CostHashMap.
inference_gateway/config.py	Added MAX_INFERENCE_ERRORS_PER_EVALUATION_RUN (defaults to 5 if not set in .env). Existing deployments won't break.
inference_gateway/main.py	Counts platform errors after each inference/embedding call. Blocks further requests with 503 once the threshold is hit. Extended /api/usage to include inference_errors and max_inference_errors. Added logger.warning() when errors are counted and when threshold blocks a request.
models/evaluation_run.py	Added PLATFORM_TOO_MANY_INFERENCE_ERRORS = 3050 in the 3xxx platform error range.
validator/main.py	After agent finishes, queries /api/usage on the inference gateway with a 10s timeout. If errors exceed the limit, marks the run as a platform error (3050) instead of scoring the patch. Also wired up the extra field in EvaluationRunException handling - it was designed but never passed through. Now agent_logs are included when reporting platform errors.
tests/test_inference_error_tracking.py (new)	19 tests covering ErrorHashMap unit behavior, platform error classification, error code validation, and integration tests against both inference and embedding gateway endpoints.

How it works end-to-end

Agent calls inference() → provider returns 500 → gateway counts error → agent gets None
Agent calls inference() → provider returns 500 → gateway counts error → agent gets None
...5th error...
Agent calls inference() → gateway returns 503 (blocked) → agent finishes with bad patch
Validator checks /api/usage → sees errors >= limit → marks run as PLATFORM error (3050)
→ Agent is not scored on this run

Config

Add to your .env if you want to override the default:

MAX_INFERENCE_ERRORS_PER_EVALUATION_RUN=5

Testing

python3 -m pytest tests/test_inference_error_tracking.py -v

[statxc]

@camfairchild Could you please review the PR? I'd appreciate any feedbacks.

[camfairchild]

Would be better termed "platform error" or something.

By "halting error" I meant one that isn't caught properly and halts the process

[camfairchild]

Looks good otherwise. Thank you

[statxc]

@camfairchild Thanks for your feedback. I updated name to platform error. Could you review again?

[statxc]

@ibraheem-abe Could you please review this PR? Welcome to any feedbacks.
thanks

[statxc]

please give me any feedbacks

[ibraheem-abe]

@statxc
Currently the entire evaluation gets restarted if we have failed inferences greater than the threshold.

We want to change this so only that specific test is retried instead of the entire thing

[statxc]

Hey @ibraheem-abe, thanks for the feedback! I think a single-run retry is out of scope for this PR. #331 was about detecting platform-side inference errors and stopping early, and this PR addresses that.

Implementing retries from our side is difficult: the platform state machine is one-way (once a run reaches running_agent, it can't go back to initializing_agent), and the validator can't create new run IDs. So retrying a single run would require changes to platform-side scheduling.

As an interim fix, I could update the SQL view so error 3050 doesn't fail the entire evaluation - failed runs would score 0 and the other results would be preserved. But I think the actual retry behavior is better handled in a separate PR.

[statxc]

@ibraheem-abe Sorry to bother you again. I'd appreciate you approve this PR if it has no problem.

[ibraheem-abe]

@statxc
I would like to test and review this with a working dev environment to ensure we don't miss or induce any edge cases.
Thank you for your patience!