Add a TESTING.md documenting how to run azlin tests.#926
Add a TESTING.md documenting how to run azlin tests.#926
Conversation
PR ReviewPR: Add a TESTING.md documenting how to run azlin tests
Summary of Changes (12 files, +647/-391)
TESTING.md Quality AssessmentStrengths:
Observations:
scripts/test_testing_docs.sh AssessmentWell-structured validation script with 14 check categories. Reads TESTING.md once to avoid 50+ grep subprocess spawns — good performance consideration. Exit code 1 on drift is CI-friendly. One observation: The script uses Hook Files AssessmentThe Observation: The hook scripts reference AGENTS.md CleanupThe -388 line deletion removes a large auto-routing prompt block. This reduces duplication and noise in the file, keeping only the essential skill invocation instructions. RecommendationMerge — this is a clean documentation PR with good coverage. The addition of Action Items:
|
Code Review — PR #926Overall Assessment: Needs Work (fixable issues, core content is solid) User Requirements Check
Issue 1: Test group counts are wrong (HIGH)Location: TESTING.md lines 28-29, 39 The doc claims 70 unit test groups and 15 handler test groups, but the actual counts are 69 and 14. The commit ca872f5 ("fix: correct test group counts") was supposed to fix this but introduced the wrong numbers. Fix: Update TESTING.md:
Issue 2: Framework artifacts committed (CRITICAL)Location: Multiple files The diff includes files that should NOT be in this PR:
Fix: Reset these files to main. Only TESTING.md and scripts/test_testing_docs.sh belong in this PR. Issue 3: Commits not squashed (LOW)Impact: Low — 3 commits should be 1 clean commit Strengths
Philosophy Compliance
SummaryThe documentation content is high quality. Two blocking issues:
|
PR Review: Documentation and ToolingPR Type: Documentation + Tooling (not a dependency update) SummaryThis PR adds a root-level Files changed (12):
TESTING.md Quality AssessmentThe new guide covers all major test categories:
The validation script ( Outside-in test results (from PR body):
Observations
RecommendationMerge — the documentation is accurate (validated by outside-in testing), the validation script reduces drift risk, and there are no dependency or security concerns. Before merging, confirm:
|
🔐 Security Review — PR #926Reviewer: Security Agent (Step 17c) Checklist
Detailed Analysis
Informational (non-blocking)
Neither finding requires action for security approval. |
🧘 Philosophy Guardian Review: PR #926 — TESTING.mdPhilosophy Score: B+The intended deliverable (TESTING.md + validation script) is philosophy-aligned. Two issues prevent an A. Strengths ✓
Concerns ⚠
Forbidden Pattern Violations ✗✗
Violations ✗
Recommendations
Regeneration AssessmentCan AI rebuild this module?
Philosophy Compliance Checklist
Overall: Strong philosophy alignment on the deliverable itself. Fix the two data accuracy issues and clean the diff to achieve full compliance. |
rysweet
force-pushed
the
feat/task-unnamed-1774884381
branch
from
ca872f5 to
266c336
Compare
Adds a top-level TESTING.md covering all test categories, commands, and environment setup. Includes a validation script (scripts/test_testing_docs.sh) that catches documentation drift by checking test counts, file references, and linked docs against the actual codebase. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rysweet
force-pushed
the
feat/task-unnamed-1774884381
branch
from
f7cfe17 to
763a176
Compare
Under set -euo pipefail, grep pattern extraction lines would cause silent script termination if TESTING.md wording changed. Now uses || true with empty-value checks to report failures via the pass/fail counters instead of crashing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ready for Final ReviewAll workflow steps completed:
Ready for merge approval. |
1 participant
Summary
Add a TESTING.md documenting how to run azlin tests.
Issue
Closes #925
Changes
{"components":[{"action":"create","name":"TESTING.md","purpose":"Root-level test running guide covering all ~3,000 tests"}],"files_to_change":[],"implementation_order":["1. Create TESTING.md with Quick Start, unit/handler/integration/E2E/agentic sections","2. Verify no secrets leaked in documentation","3. Commit"],"new_files":["TESTING.md"],"risks":["Test counts (68 unit groups, 13 handler groups, ~3,000 total) may drift as code evolves — periodic refresh needed"],"security_considerations":["Doc references env var names (ANTHROPIC_API_KEY) but never actual values — verified clean"],"test_files":[]}
Testing
Checklist
This PR was created as a draft for review before merging.
Step 16b: Outside-In Testing Results
Scenario 1 — Verify
cargo test --allmatches documented countsCommand:
cd rust && cargo test --all 2>&1 | grep "^test result:"Result: PASS
Output: 2,996 passed; 0 failed; 114 ignored — matches updated TESTING.md claims (~3,000 / ~114)
Scenario 2 — Verify all linked documentation files exist
Command:
for f in <8 doc paths>; do test -f "$f" && echo EXISTS; doneResult: PASS
Output: All 8 linked doc files exist (TEST_SUITE_SPECIFICATION.md, AGENTIC_INTEGRATION_TESTS.md, etc.)
Scenario 3 — Verify integration test file listing accuracy
Command:
ls rust/crates/azlin/tests/*.rs | wc -lResult: PASS
Output: 18 files found, matching all 18 files listed in TESTING.md table
Scenario 4 — Verify .cargo/config.toml RUST_MIN_STACK claim
Command:
cat rust/.cargo/config.tomlResult: PASS
Output:
RUST_MIN_STACK = "8388608"confirmedFix iterations: 1 (corrected test group counts: 69→70, 14→15; total
2,950→3,000)