fix(bmad-product-brief): tighten update/validate rules and eval expectations#2371
fix(bmad-product-brief): tighten update/validate rules and eval expectations#2371
Conversation
…tations - Update: audit trail (decision-log + addendum) is now mandatory before modifying brief.md in headless mode; distillate regeneration is required - Validate: always emits offer_to_update in headless JSON output - Headless: added validate example to JSON status block docs - Evals B8: add 900s timeout, replace hard 30%-smaller check with meaningful-condensation expectation - Eval B9: sharpen right-sized expectation wording
🤖 Augment PR SummarySummary: Tightens headless Update/Validate behavioral requirements for 🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
| { | ||
| "id": "B8", | ||
| "_pattern": "process-discipline", | ||
| "timeout": 900, |
There was a problem hiding this comment.
evals/bmm-skills/bmad-product-brief/evals.json:241 Adding a per-eval timeout is useful, but can you confirm the eval runner actually supports a timeout field here (and that 900 is interpreted as seconds, not ms) so it won’t be ignored or fail strict schema validation?
Severity: medium
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| **Create.** A brief the user is proud of, that meets their needs, drawn out through real conversation — do not assume: instead converse and understand, and then help craft the best product brief for their needs. Begin in `## Discovery` before drafting; the brief comes after the picture is on the table. Shape follows the product and need. Treat `{workflow.brief_template}` as a starting structure, not a contract: drop sections that do not earn their place, add sections the product needs, reorder freely - create sections for specialized domains or concerns also as needed. The brief serves the product's story, not the template's shape. Bind `{doc_workspace}` to a fresh folder at `{workflow.output_dir}/{workflow.output_folder_name}/` and write `brief.md` there with YAML frontmatter (title, status, created, updated). For Update and Validate, `{doc_workspace}` is the existing folder of the brief being targeted. | ||
|
|
||
| **Update.** Reconcile an existing brief with a change signal (edit request, downstream artifact, anything). Read the brief, the addendum if present, `decision-log.md`, and any original inputs first — past decisions and rejected ideas matter. Then run the `## Discovery` posture against the change signal before proposing changes. Identify what is now stale or wrong, propose changes, apply on agreement, bump `updated`. If the change signal contradicts prior decisions, surface the conflict before changing anything. In headless mode, if the prompt clearly signals intent to override the contradicted decision, proceed with the change and autonomously write the full audit trail — a new `decision-log.md` entry naming the reversal and its rationale, and an `addendum.md` override section — without waiting for user confirmation; if intent to override is ambiguous, halt with `blocked` status naming the specific conflict. If the change is fundamental, name it as a re-draft and offer Create instead. If `distillate.md` exists, regenerate it after changes are applied (re-invoke `bmad-distillator` per Finalize step 3); if unavailable, flag the distillate as stale. | ||
| **Update.** Reconcile an existing brief with a change signal (edit request, downstream artifact, anything). Read the brief, the addendum if present, `decision-log.md`, and any original inputs first — past decisions and rejected ideas matter. Then run the `## Discovery` posture against the change signal before proposing changes. Identify what is now stale or wrong, propose changes, apply on agreement, bump `updated`, and write a new `decision-log.md` entry recording what changed and why — every update, clean or override, must be logged. If the change signal contradicts prior decisions, surface the conflict before changing anything. In headless mode, if the prompt clearly signals intent to override the contradicted decision, write the full audit trail first, then apply the change — you must: (1) add a new entry to `decision-log.md` naming the decision being reversed and its rationale, (2) add an override section to `addendum.md` (creating it if absent). Both are mandatory before modifying `brief.md`; do not wait for user confirmation. If intent to override is ambiguous, halt with `blocked` status naming the specific conflict. If the change is fundamental, name it as a re-draft and offer Create instead. If `distillate.md` exists, you must regenerate it after changes are applied by invoking `bmad-distillator`; this step is required, not optional. If `bmad-distillator` is unavailable, flag the distillate as stale in the JSON output. |
There was a problem hiding this comment.
src/bmm-skills/1-analysis/bmad-product-brief/SKILL.md:32 This says to “flag the distillate as stale in the JSON output” if bmad-distillator is unavailable, but the ## Finalize section later specifies a different JSON contract for that case; the mixed guidance could lead to inconsistent headless JSON output.
Severity: medium
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| **Update.** Reconcile an existing brief with a change signal (edit request, downstream artifact, anything). Read the brief, the addendum if present, `decision-log.md`, and any original inputs first — past decisions and rejected ideas matter. Then run the `## Discovery` posture against the change signal before proposing changes. Identify what is now stale or wrong, propose changes, apply on agreement, bump `updated`, and write a new `decision-log.md` entry recording what changed and why — every update, clean or override, must be logged. If the change signal contradicts prior decisions, surface the conflict before changing anything. In headless mode, if the prompt clearly signals intent to override the contradicted decision, write the full audit trail first, then apply the change — you must: (1) add a new entry to `decision-log.md` naming the decision being reversed and its rationale, (2) add an override section to `addendum.md` (creating it if absent). Both are mandatory before modifying `brief.md`; do not wait for user confirmation. If intent to override is ambiguous, halt with `blocked` status naming the specific conflict. If the change is fundamental, name it as a re-draft and offer Create instead. If `distillate.md` exists, you must regenerate it after changes are applied by invoking `bmad-distillator`; this step is required, not optional. If `bmad-distillator` is unavailable, flag the distillate as stale in the JSON output. | ||
|
|
||
| **Validate.** Honest critique against the brief's own purpose. Read the brief, the addendum if present, `decision-log.md`, and any original inputs first — a validation that ignores prior decisions, rejected ideas, or context the user supplied is shallow. Cite specific lines. Caveat what cannot be evaluated. Return inline — no separate file unless asked. Offer to roll findings into an Update. | ||
| **Validate.** Honest critique against the brief's own purpose. Read the brief, the addendum if present, `decision-log.md`, and any original inputs first — a validation that ignores prior decisions, rejected ideas, or context the user supplied is shallow. Cite specific lines. Caveat what cannot be evaluated. Return inline — no separate file unless asked. Always offer to roll findings into an Update, even in headless mode — include `"offer_to_update": true` in the JSON status block. |
There was a problem hiding this comment.
src/bmm-skills/1-analysis/bmad-product-brief/SKILL.md:34 Since headless Validate now requires "offer_to_update": true in the JSON status block, consider adding an eval expectation (e.g. A4) that asserts this key is present so the new contract can’t silently regress.
Severity: low
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
|
Caution Review failedPull request was closed or merged during review 📝 WalkthroughWalkthroughThis PR updates evaluation criteria and operating procedures for the BMAD Product Brief skill. B8 distillate expectations shift from quantitative size reduction to semantic quality (meaningful condensation without near-verbatim copying). C1 brief sizing explicitly disallows enterprise-credibility padding for bootstrapped scenarios. Corresponding SKILL.md changes enforce audit trails (decision-log.md, addendum.md), require distillate regeneration after Updates, and add explicit offer-to-update flags in Validate/Headless JSON responses. ChangesBMAD Product Brief Evaluation & Skill Procedure Updates
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1 participant
Summary
brief.md; distillate regeneration is required, not optional"offer_to_update": truein JSON status block; added validate example to headless docsTest plan
offer_to_update: true