Add AI visual regression tester via GitHub Agentic Workflows#811
Add AI visual regression tester via GitHub Agentic Workflows#811supervoidcoder merged 4 commits intomainfrom
Conversation
Co-authored-by: supervoidcoder <88671013+supervoidcoder@users.noreply.github.com>
|
@copilot are you stupid why do you think i sent you that discussion link ... sighh look at this thread: so look at the unbloat docs md. you will see that AGENTIC WORKFLOWS ALREADY COMES WITH PLAYWRIGHT TOOL AND IT SHOULDN'T USE CLI |
Co-authored-by: supervoidcoder <88671013+supervoidcoder@users.noreply.github.com>
... Fixed in d4e42cf. The workflow now uses the Playwright MCP tool directly — the agent calls Playwright to navigate URLs and take screenshots (saved automatically to Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
🤖 Auto-Component Test Results🎉 No new UI components detected!This PR doesn't add any new buttons, menus, or clickable components. Triggered by Copilot • Commit |
📝 Mega PR Test SkippedHi @Copilot! The comprehensive Mega PR test was skipped because this PR only changes documentation or configuration files (.md, .txt, .yml, .yaml). To force the Mega PR test to run, comment |
🎬 Visual Regression Testing - SkippedHi @Copilot! VRT was skipped because this PR only changes documentation or configuration files (.md, .txt, .yml). To force VRT to run, comment |
There was a problem hiding this comment.
Pull request overview
Adds a new GitHub Agentic Workflow to perform AI-assisted visual regression checks on UI-related pull requests, replacing the existing brittle static visual regression approach.
Changes:
- Introduces a human-editable gh-aw workflow source that builds the app, drives UI navigation via Playwright MCP, and reports screenshot comparisons.
- Adds the compiled
.lock.ymlworkflow that implements gating (org roles, non-draft, concurrency) and runs the Copilot-based agent with safe outputs and cache-memory baseline persistence.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
.github/workflows/ai-visual-regression.md |
Defines the agentic workflow frontmatter (gating, tool allowlist, cache-memory) and the agent prompt (diff analysis → build → serve → screenshot → baseline compare → report). |
.github/workflows/ai-visual-regression.lock.yml |
Compiled GitHub Actions workflow generated by gh aw compile, including activation/pre-activation gating, agent execution, safe-outputs comment posting, and cache-memory persistence. |
| - "ls *" | ||
| - "echo *" | ||
| - "kill *" | ||
| - "curl *" |
There was a problem hiding this comment.
The prompt’s health-check loop uses seq and sleep, but those commands aren’t allowed in the tools.bash allowlist. As-is, the agent will likely be blocked when following Step 3. Either add "sleep *" (and "seq *", if keeping this exact loop) to tools.bash, or change the polling example to avoid those commands (e.g., brace expansion plus a Playwright/Node-based wait).
| - "curl *" | |
| - "curl *" | |
| - "sleep *" | |
| - "seq *" |
| safe-outputs: | ||
| add-comment: | ||
| max: 1 | ||
| hide-older-comments: true |
There was a problem hiding this comment.
The safe-outputs add-comment tool causes the compiled workflow to request discussions: write permission, which appears unnecessary for a PR-only reporter. Consider setting safe-outputs.add-comment.discussions: false (per the tool’s documented option) so the workflow runs with least privilege while still being able to comment on PRs/issues.
| hide-older-comments: true | |
| hide-older-comments: true | |
| discussions: false |
| const allowedExtensions = [".png",".json"]; | ||
| const result = validateMemoryFiles('/tmp/gh-aw/cache-memory', 'cache', allowedExtensions); | ||
| if (!result.valid) { | ||
| core.setFailed(`File type validation failed: Found $${result.invalidFiles.length} file(s) with invalid extensions. Only .png, .json are allowed.`); |
There was a problem hiding this comment.
Minor bug in the failure message template: the string interpolation uses $${result.invalidFiles.length} which will render an extra $ (e.g., Found $3 file(s) ...). This should be ${result.invalidFiles.length} like the similar step later in the file, otherwise the error output is confusing when file type validation fails.
| core.setFailed(`File type validation failed: Found $${result.invalidFiles.length} file(s) with invalid extensions. Only .png, .json are allowed.`); | |
| core.setFailed(`File type validation failed: Found ${result.invalidFiles.length} file(s) with invalid extensions. Only .png, .json are allowed.`); |
There was a problem hiding this comment.
you're not supposed to edit the compiled yml
|
@copilot apply changes based on the comments in this thread (don't edit the compiled file directly) don't forget to recompile |
…ermission Co-authored-by: supervoidcoder <88671013+supervoidcoder@users.noreply.github.com>
Done in 7e2cdcb:
Skipped the Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
.github/workflows/ai-visual-regression.mdagentic workflow"sleep *"and"seq *"to bash allowlist so the health-check polling loop can rundiscussions: falsetosafe-outputs.add-commentto drop unnecessarydiscussions: writepermission (least privilege).github/workflows/ai-visual-regression.lock.yml(viagh aw compile)Original prompt
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.