revert: remove redundant retry functionality #5720
Conversation
Remove retry command and --retry-errors flag as they are redundant with existing --filter-errors-only capability. The retry functionality provided two ways to re-run failed tests: 1. promptfoo retry <evalId> 2. promptfoo eval --retry-errors However, the same outcome can be achieved more simply with: promptfoo eval --filter-errors-only <evalId> The existing filter approach is preferable because: - Simpler implementation without database manipulation - No risk of corrupting evaluation data - Preserves evaluation history - Same practical result (re-running error tests) Reverted changes from commit ab6ae2b: - Removed src/commands/retry.ts - Removed --retry-errors flag and logic from eval command - Removed retry command registration from main.ts - Removed retry documentation - Removed retry testing examples - Reverted schema and type changes
📝 WalkthroughWalkthroughThis change removes the retry-errors functionality across the codebase. It deletes the retry CLI command (src/commands/retry.ts) and its wiring (src/main.ts), removes the --retry-errors option from eval (src/commands/eval.ts), and updates related schemas/types (site/static/config-schema.json, src/types/index.ts). Documentation is updated to remove the option and “Retry Errors” section (site/docs/usage/command-line.md). The retry-testing example and its files are removed (examples/retry-testing/*). Eval command logic is adjusted to rely on resumeRaw for flow control and option application, replacing prior resumeEval checks. Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests
📜 Recent review detailsConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro 📒 Files selected for processing (9)
💤 Files with no reviewable changes (8)
🧰 Additional context used📓 Path-based instructions (1)**/*.{ts,tsx}📄 CodeRabbit inference engine (.cursor/rules/gh-cli-workflow.mdc)
Files:
🧠 Learnings (1)📓 Common learnings🧬 Code graph analysis (1)src/commands/eval.ts (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (20)
🔇 Additional comments (6)
Comment |
Summary
Remove the retry command and
--retry-errorsflag functionality that was added in #5647, as it is redundant with existing filtering capabilities.Problem Analysis
The retry functionality provided two ways to re-run failed tests:
promptfoo retry <evalId>- retries ERROR results from a specific evaluationpromptfoo eval --retry-errors- retries ERROR results from latest evaluationHowever, the same practical outcome can be achieved more simply with the existing:
Why the existing approach is better
Changes Made
src/commands/retry.ts(entire retry command implementation)--retry-errorsflag and logic fromsrc/commands/eval.tssrc/main.tsexamples/retry-testing/site/static/config-schema.jsonsrc/types/index.tsQuality Assurance
npm run l && npm run f)npm run tsc)npm test)Implications
Users who were relying on the retry commands can achieve the same result with:
Instead of:
Use:
This approach provides the same functionality while being simpler, safer, and more consistent with the existing codebase architecture.