Skip to content

feat: Log failed LLM calls to PromptLayer before re-raising#295

Closed
adagradschool wants to merge 1 commit intomasterfrom
feat/error-tracking-on-llm-failure
Closed

feat: Log failed LLM calls to PromptLayer before re-raising#295
adagradschool wants to merge 1 commit intomasterfrom
feat/error-tracking-on-llm-failure

Conversation

@adagradschool
Copy link
Contributor

Summary

When .run() calls an LLM provider and the call throws (rate limit, auth, timeout, etc.), the SDK now catches the exception, logs it to PromptLayer with status=ERROR, error_type, and error_message, then re-raises. Also exposes these fields on log_request() for manual callers. Error categorization is duck-typed — no provider imports in runtime code.

Edge cases tested

  • Tracking failure doesn't mask the original LLM exception
  • Success path doesn't include error fields
  • status_code=402 quota detection
  • Generic exceptions with "quota"/"timeout" in message only match for known provider modules (not arbitrary errors)
  • Real OpenAI/Anthropic exception classes (rate limit, auth, timeout, bad request)

Known concern

When PromptLayer API is down, the error-tracking call in the except block may add latency due to retries (via track_request/atrack_request retry logic) before the original exception propagates. Consider a no-retry or shorter-timeout path for error logging.

Test plan

  • tests/test_error_categorization.py — 11 parametrized cases
  • tests/test_error_logging.py — 5 cases (sync/async run + success path)
  • Full suite passes (71 tests), ruff lint + format clean

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants