Skip to content

Validate LLM responses and warn on unexpected formats#21

Merged
jackparnell merged 1 commit intomainfrom
feature/llm-response-validation
Apr 7, 2026
Merged

Validate LLM responses and warn on unexpected formats#21
jackparnell merged 1 commit intomainfrom
feature/llm-response-validation

Conversation

@ColonistOne
Copy link
Copy Markdown
Collaborator

Summary

The agent now catches and logs when the LLM ignores the expected response format, instead of silently doing nothing.

What's validated

Vote parsing — warns when the response contains none of UPVOTE, DOWNVOTE, or SKIP:

WARNING LLM response missing vote keyword (UPVOTE/DOWNVOTE/SKIP) for 'Some post': I think this post is really interesting...

Comment extraction — improved parsing:

  • Explicit COMMENT: SKIP now correctly returns empty (was falling through)
  • SKIP anywhere in response treated as skip intent
  • Short ambiguous responses (<20 chars) logged as warning
  • Freeform comments (no COMMENT: prefix) logged at debug level

Reply to comments — rejects replies under 10 characters:

WARNING LLM reply to alice too short to post (7 chars): 'Thanks!'

Prevents posting low-effort single-word replies.

Test plan

  • Missing vote keyword → warning logged, no vote cast
  • Vote-only response → no comment extracted
  • SKIP in middle of text → treated as skip
  • Explicit COMMENT: SKIP → empty comment
  • Short reply rejected with warning
  • 163 tests pass, ruff + mypy clean

🤖 Generated with Claude Code

The agent now logs warnings when the LLM ignores the expected
response format instead of silently falling through:

- Vote parsing: warns when response contains neither UPVOTE,
  DOWNVOTE, nor SKIP (LLM ignored the instruction)
- Comment extraction: debug log for freeform format (no COMMENT:
  prefix), warning for ambiguous short responses
- Reply to comments: rejects replies under 10 chars with warning
  (prevents posting "ok" or "Thanks!" as a reply)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jackparnell jackparnell merged commit 3ab5fad into main Apr 7, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants