-
Notifications
You must be signed in to change notification settings - Fork 1
Description
Situation
nabledge-test baseline run (20260312-155314) shows qa-004 detection rate at 4/7 (57.1%), the lowest among all scenarios. The three undetected keywords are TransactionAbnormalEnd, ProcessAbnormalEnd, and ResumeDataReader — all specific to batch error handling.
Previous baseline (20260307-144233) had qa-004 at 7/7 (100%), so this is a regression introduced by the full knowledge file regeneration in v0.5.
Root cause hypothesis: the knowledge-creator generation/fix prompts do not sufficiently instruct the model to include error handling class names (TransactionAbnormalEnd, ProcessAbnormalEnd, ResumeDataReader) in the knowledge files for batch error handling sections. As a result, these class names are either missing or buried in the regenerated files, making them unreachable by knowledge search.
Reference: .pr/00169/nabledge-test/report-202603121553.md — 仮説1
Pain
Users asking "バッチのエラーハンドリングはどうすればいいですか?" receive answers that do not mention the key classes (TransactionAbnormalEnd, ProcessAbnormalEnd, ResumeDataReader), making the answer incomplete and potentially misleading.
Benefit
- Users can get correct and complete batch error handling guidance including the specific exception classes to use
Success Criteria
- Root cause identified: confirm which knowledge files are missing the target class names
- knowledge-creator generation or fix prompt updated to ensure these class names appear in the appropriate sections
- qa-004 detection rate recovers to ≥ 6/7 in a nabledge-test run after the fix
- Comparison report generated to confirm improvement vs 20260312-155314 baseline