feat: consolidate vnext workstream (supersedes #12)#15
Merged
digitarald merged 106 commits intomicrosoft:mainfrom Feb 24, 2026
Merged
feat: consolidate vnext workstream (supersedes #12)#15digitarald merged 106 commits intomicrosoft:mainfrom
digitarald merged 106 commits intomicrosoft:mainfrom
Conversation
…ing and CLI options
feat(eval): implement model listing functionality in eval command feat(copilot): create function to extract and list Copilot models from CLI help feat(evaluator): enhance trajectory viewer with phase filtering for tool calls feat(tui): add readiness report feature with detailed output and user interaction
…text - Added a systemMessage field to primer.eval.json to clarify response context. - Updated evalScaffold.ts to include instructions for incorporating the system message in generated cases. - Modified evaluator.ts to use a default system message when none is provided, ensuring responses are relevant to the repository.
…d enhance TUI - Removed the analyze command from the CLI as it is no longer needed. - Updated the Primer evaluation workflow to include timestamped output files and improved error handling. - Enhanced the TUI to manage multiple Copilot models for evaluation and judging, allowing users to cycle through available models. - Improved user prompts and messages throughout the TUI for better clarity and user experience. - Adjusted the structure of the evaluation results display and added readiness report functionality.
- Fix TypeScript syntax errors in tui.tsx and github.ts - Create visual report generator service with beautiful HTML output - Add --visual flag to readiness command for HTML reports - Implement batch-readiness command for multi-repo visual reports - Add BatchReadinessTui for interactive repository selection - Support both GitHub and Azure DevOps repository sources - Include summary cards, pillar performance, and level distribution charts - Update documentation with visual report examples Co-authored-by: pierceboggan <1091304+pierceboggan@users.noreply.github.com>
- Added support for selecting and running evaluations with a new eval-pick status. - Implemented batch processing options for GitHub and Azure DevOps. - Introduced a logging mechanism to track activity and status updates. - Enhanced user interface with spinner animations and improved status indicators. - Refactored code to check for eval configuration on mount and display relevant messages. - Updated command hints for better user guidance during interactions. - Removed unused readiness report functionality and related types.
…rt generation - Implement tests for `ensureDir` and `safeWriteFile` in `fs.test.ts`. - Create comprehensive tests for `runReadinessReport` in `readiness.test.ts`, covering various criteria and pillars. - Add tests for `generateVisualReport` in `visualReport.test.ts`, ensuring correct HTML output and content. - Update `generateCopilotInstructions` to use a new preferred model. - Enhance TUI to support model selection and generation options for copilot instructions and agents. - Introduce `tsup` configuration for building the project.
…eport Add visual AI readiness reports with batch processing
…ove error handling
… a prioritized fix list
- Added support for additional languages (C#, Java, Ruby, PHP) in the analyzeRepo function. - Updated detectPackageManager to recognize new package managers (Maven, Gradle, Bundler, Composer). - Improved handling of pnpm workspace files by skipping comment-only lines and supporting inline arrays. fix: validate Azure DevOps slugs and improve error handling - Introduced validateAdoSlug function to ensure organization, project, and repo names are valid. - Updated API calls in Azure DevOps service to use validated slugs. - Enhanced error handling in checkRepoHasInstructions to throw descriptive errors on request failures. refactor: streamline Copilot CLI path resolution - Cached Copilot CLI path to avoid redundant lookups. - Improved findCopilotCliPath to handle platform-specific paths more effectively. - Added glob pattern matching for VS Code extension paths. chore: update evalScaffold and evaluator services - Refactored generateEvalScaffold to use withCwd for better directory management. - Simplified runEval by using assertCopilotCliReady for CLI path resolution. - Removed redundant EvalCase and EvalConfig type definitions from evaluator. fix: sanitize error messages in git push - Added error handling in pushBranch to sanitize embedded credentials from error messages. feat: enhance instruction generation and PR body creation - Updated generateCopilotInstructions to use withCwd for improved directory handling. - Created utility functions for building PR bodies for configurations and instructions. - Updated BatchTui and BatchTuiAzure to use DEFAULT_MODEL for instruction generation. chore: add utility functions for file system operations - Introduced validateCachePath to prevent path traversal vulnerabilities. - Enhanced safeWriteFile to reject symlinks and ensure safe file writing.
…ve workspace detection - Added support for detecting non-JS monorepos (Cargo, Go, .NET, Gradle, Maven). - Updated RepoApp and RepoAnalysis types to include ecosystem and manifestPath. - Improved workspace type detection to accommodate additional ecosystems. - Refactored app resolution logic to handle non-JS monorepos when JS apps are insufficient. fix(azureDevops): encode memberId in accounts URL - Updated accounts URL construction to properly encode memberId. refactor(copilot): implement caching for CLI path resolution - Added caching mechanism for Copilot CLI path with a TTL of 5 minutes. - Improved path resolution logic to handle different platforms. refactor(evalScaffold): simplify progress callback type - Updated onProgress callback type to use a more concise parameter. refactor(evaluator): define CopilotClient and CopilotSession interfaces - Introduced interfaces for better type safety and clarity in Copilot session management. refactor(instructions): enhance prompt for generating Copilot instructions - Updated prompt to clarify analysis requirements and include additional tech stack files. refactor(readiness): streamline readiness checks with improved status handling - Refactored readiness criteria checks to return structured status and reason. fix(ui): handle errors during repo loading and processing - Added error handling for repo loading and processing in BatchTui and BatchTuiAzure components. chore(tui): improve error logging for repo analysis failures - Enhanced error logging to provide clearer feedback on repo analysis issues. docs(cwd): add warning about process.chdir() side effects - Updated documentation to clarify the implications of using process.chdir(). refactor(fs): export utility functions for file system operations - Made fileExists, safeReadDir, and readJson functions public for broader usage.
… tests refactor(analyzer): normalize package.json and Cargo.toml paths in workspace resolution refactor(copilot): normalize CLI path when found in Copilot CLI path resolution
…prove error reporting
…used-vars, fix unused imports, auto-fix import order
…ing, dead stubs, typo
…oncurrency control
TypeScript's compiler handles undeclared variable checks natively. Disabling ESLint's no-undef prevents false positives for Node.js globals (process, console, setTimeout, fetch, Buffer) per typescript-eslint recommendation.
Dogfood improvements: CLI UX, extension polish, safeWriteFile, docs split
…R service, and UX polish
fix(vscode): harden extension commands with overwrite flows, shared PR service, and UX polish
refactor: consolidate PR file handling
…t ordering - isAllowedSystemAlias() now returns true on win32 since lstat already confirmed no symlinks; realpath differences are just 8.3/case normalization - Fix import ordering in 7 files to satisfy import-x/order lint rule
- Replace blanket isAllowedSystemAlias win32 return with ancestor lstat walk to correctly distinguish 8.3 normalization from real symlinks - Fix policy loadPolicy catch to handle cross-realm errors (instanceof Error fails across vitest worker threads)
Rewrite Readiness with Signal Engine
Introduce a new plugin-based policy engine that runs alongside the existing readiness system in shadow mode for safe validation. New modules in src/services/policy/: - types.ts: Signal, Recommendation, Grade, PolicyContext interfaces - compiler.ts: compiles policy configs into executable plugin chains - loader.ts: loads and validates plugin chains from policy sources - engine.ts: executes plugins, computes scores and grades - adapter.ts: bridges engine reports to legacy ReadinessReport format - shadow.ts: shadow mode logging for comparison with legacy path Changes to existing code: - readiness.ts: adds optional 'shadow' mode and 'engine' field on ReadinessReport; refactors isConfigSourced tracking for clarity - utils/fs.ts: clarifies validateCachePath caller responsibilities - README.md: documents new policy/ directory and links plugin guide Includes comprehensive test coverage (7 new test files, all 440 tests pass) and plugin authoring documentation in docs/plugins.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
feat: add plugin-based policy engine with shadow mode
5 tasks
added 10 commits
February 22, 2026 15:27
This was referenced Feb 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Explainer
This PR is the consolidated "vnext" integration from
digitarald/primerinto upstreammain.The goal is to land one coherent, reviewable change set instead of several fragmented PRs. It combines CLI work, VS Code extension parity updates, policy/readiness improvements, and repository workflow/docs hardening.
Why This Exists
What Changed (High Level)
src/commands/*andsrc/cli.ts.src/services/*(readiness, evaluator, analyzer, policy modules, Git/GitHub/Azure paths).vscode-extension/src/*for command UX and integration flow.src/services/__tests__/*..github/*(CI/eval/release workflows, templates, prompts, agent files).README.md,CONTRIBUTING.md,examples/*,PLAN.md).Reviewer Guide
Start here for fastest signal:
.github/workflows/ci.ymland repo config changessrc/services/policy/*andsrc/services/readiness.tssrc/commands/*integration pointsvscode-extension/src/commands/*+vscode-extension/src/services.tssrc/services/__tests__/*Risk and Compatibility
Validation
npm run lintnpm run format:check(tracked files)npm run typechecknpm run testnpm run buildcd vscode-extension && npx tsc --noEmit && node esbuild.mjsSupersedes