Skip to content

Expand scope-req-audit into adversarial requirements validation and consolidate workflows #144

@aireilly

Description

@aireilly

Summary

  1. Add an adversarial review pass to scope-req-audit — after code classification, a second agent re-reads the original JIRA ticket, PRs, and specs, then flags requirements that were missed, hallucinated, or inaccurately extracted. Uses the existing code evidence to strengthen its checks.

  2. Consolidate the two workflow YAMLs into a single docs-workflow.yaml. Code-dependent steps already have when: has_source_repo — no need for two files.

Adversarial review pass

What changes

After the existing fanout (phase 1 — code classification, unchanged), dispatch one additional agent that:

  1. Re-reads the JIRA ticket, PR diffs, and any linked specs using the same scripts the discoverer used (jira_reader.py, git_pr_reader.py)
  2. Compares them against requirements.md
  3. Produces a short findings list: missed requirements, hallucinated requirements, and inaccurate acceptance criteria
  4. When code evidence is available, cross-references findings against evidence-status.json and api-surface.json

The reviewer does NOT see the analyst agents' intermediate reasoning — only the final requirements.md. This is the adversarial element: an independent agent checking the work with fresh eyes and access to the same sources.

What it catches

  • A PR adds a new CLI flag but no REQ covers it
  • An analyst fabricated a requirement not mentioned in any JIRA ticket or PR
  • Acceptance criteria state wrong limits, wrong flag names, or wrong behavior vs. what the JIRA ticket says
  • A requirement conflicts with another requirement

Output

Findings appended to evidence-status.json as a new review_findings array:

{
  "review_findings": [
    {
      "type": "missed|hallucinated|inaccurate|conflicting",
      "requirement": "REQ-005",
      "description": "REQ-005 describes webhook notifications — not mentioned in any JIRA ticket or PR.",
      "severity": "high",
      "remediation": "Remove or confirm with stakeholder"
    }
  ]
}

All existing fields unchanged — additive only.

Step-result.json extension

Two new optional fields:

{
  "review_finding_count": 3,
  "review_has_high_severity": true
}

The orchestrator checks review_has_high_severity — if true, pauses and presents findings to the user before continuing to planning.

When no source repo

Remove the when: has_source_repo condition from scope-req-audit. Without a repo, skip phase 1 (code classification) and run only the adversarial review pass against JIRA/PR/spec sources. Requirements get "status": "unclassified" instead of grounded/partial/absent.

Workflow consolidation

Merge docs-workflow.yaml and docs-workflow-code-evidence.yaml into one file:

workflow:
  name: docs-workflow

steps:
  requirements:
    skill: docs-tools:docs-workflow-requirements

  scope-req-audit:
    skill: docs-tools:docs-workflow-scope-req-audit
    inputs: [requirements]

  planning:
    skill: docs-tools:docs-workflow-planning
    inputs: [requirements, scope-req-audit]

  code-evidence:
    skill: docs-tools:docs-workflow-code-evidence
    when: has_source_repo
    inputs: [planning]

  writing:
    skill: docs-tools:docs-workflow-writing
    inputs: [planning, code-evidence]

  technical-review:
    skill: docs-tools:docs-workflow-tech-review
    inputs: [writing, code-evidence]

  style-review:
    skill: docs-tools:docs-workflow-style-review
    inputs: [writing]

  create-merge-request:
    skill: docs-tools:docs-workflow-create-merge-request
    when: create_merge_request
    inputs: [writing, style-review, technical-review]

scope-req-audit runs unconditionally. code-evidence is the only step gated on has_source_repo. Deprecate --workflow code-evidence and docs-workflow-code-evidence.yaml.

Implementation

  1. Add adversarial review agent dispatch to scope-req-audit SKILL.md after the existing merge step
  2. Extend evidence-status.json with review_findings array
  3. Extend step-result.json with review_finding_count and review_has_high_severity
  4. Remove when: has_source_repo from scope-req-audit — handle missing repo internally (skip code classification, still run review)
  5. Merge workflow YAMLs into single docs-workflow.yaml
  6. Deprecate docs-workflow-code-evidence.yaml
  7. Update orchestrator to pause on review_has_high_severity: true
  8. Verify downstream steps handle "status": "unclassified" (treat as include, no code evidence)

Note on item 8: Currently the planning step checks whether evidence-status.json exists — if missing (step was skipped), it includes all requirements with no code evidence signal. With the consolidated workflow, evidence-status.json will always exist (scope-req-audit always runs), but without a repo the requirements will have "status": "unclassified" instead of grounded/partial/absent. The planning step needs to treat unclassified the same as "file not found" — include the requirement, no code evidence signal. Same applies to the writing step's grounded/partial/absent handling.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions