Run unit tests only for plugins with changes

[ShyamGadde]

Summary

Fixes #1792

Relevant technical choices

• Run unit tests only for the plugin(s) affected by the changes in the pull request.
• Changes to the Optimization Detective plugin trigger tests for Embed Optimizer and Image Prioritizer due to their dependencies.
• All tests will run for all plugins when commits are added to the trunk branch.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.17%. Comparing base (5db32da) to head (e9596f9).
⚠️ Report is 8 commits behind head on trunk.

Additional details and impacted files

@@           Coverage Diff           @@
##            trunk    #1838   +/-   ##
=======================================
  Coverage   69.17%   69.17%           
=======================================
  Files          90       90           
  Lines        7708     7708           
=======================================
  Hits         5332     5332           
  Misses       2376     2376           

Flag	Coverage Δ
auto-sizes	93.38% <ø> (?)
dominant-color-images	87.02% <ø> (?)
embed-optimizer	67.34% <ø> (?)
image-prioritizer	90.35% <ø> (?)
multisite	?
optimization-detective	95.04% <ø> (?)
performance-lab	53.62% <ø> (?)
single	?
speculation-rules	71.25% <ø> (?)
view-transitions	9.70% <ø> (?)
web-worker-offloading	1.27% <ø> (?)
webp-uploads	62.85% <ø> (?)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ShyamGadde]

This limits the directory output to a maximum depth of 2. For example, plugins/<plugin-name>/tests will be returned as plugins/<plugin-name>.
Since dir_names: true ensures only unique directories are listed, combining it with dir_names_max_depth: 2 allows us to extract a clean list of changed plugin names. This avoids additional logic in the next step for determining modified plugins.

[ShyamGadde]

Currently, optimization-detective is the only plugin with dependencies, so we could simplify this by checking for it explicitly instead of maintaining a general dependency structure. However, I think this approach keeps things flexible if more dependencies are introduced in the future.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: ShyamGadde <shyamgadde@git.wordpress.org>
Co-authored-by: westonruter <westonruter@git.wordpress.org>
Co-authored-by: felixarntz <flixos90@git.wordpress.org>
Co-authored-by: swissspidy <swissspidy@git.wordpress.org>
Co-authored-by: thelovekesh <thelovekesh@git.wordpress.org>
Co-authored-by: adamsilverstein <adamsilverstein@git.wordpress.org>
Co-authored-by: phanduynam <phanduynam@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[westonruter]

PhpStorm (via Shellcheck) is flagging this line:

Arrays implicitly concatenate in [[ ]]. Use a loop (or explicit * instead of @).
See SC2199.

[ShyamGadde]

Fixed it in 11949ac.

[felixarntz]

@ShyamGadde This looks great to me so far, just one minor comment.

Before we merge this, I would like to discuss though under which conditions we should do this. To me, it makes sense to only run selected unit tests for PRs, to speed up the process. But running the whole test suite also has value, e.g. to warn us early in case something breaks somewhere else where we didn't change any files - which can happen e.g. based on WordPress Core updates.

How about we implement this change so that it only applies to pull requests, but for merges to our main branches it would still run all test suites? IMO this would be the best of both worlds:

• Running tests against PRs would be more efficient, faster and cost less resources.
• Running tests against code merged into trunk and release branches would still be ensured to be more comprehensively tested.

It probably wouldn't be a huge lift to change that, we could make the step that determines the changed files and plugins conditional to pull_request events and otherwise return the list of all plugins.

WDYT? cc @westonruter @thelovekesh

[felixarntz]

Maybe call this PLUGIN_DEPENDENTS to clarify? In this case, Optimization Detective is a dependency of the other two, but the other two are dependents of Optimization Detective.

[ShyamGadde]

That makes sense. Updated it in 90a5ab7.

[ShyamGadde]

I noticed that the threshold for patch coverage is set to 80%, which (based on Codecov’s docs) means that coverage can drop by up to 80% while still passing. I'm wondering if this was intentional—perhaps it was meant to ensure a minimum of 80% coverage rather than allowing such a large drop?

Additionally, since target isn't explicitly set, it defaults to the project coverage from the pull request base. That results in PR checks displaying something like:

Would it make sense to set target: 100% for patch coverage, with a lower threshold (maybe 10-20%), to ensure new changes are reasonably covered? Just wanted to check if this aligns with the intended behavior.

[swissspidy]

I'm wondering if this was intentional—perhaps it was meant to ensure a minimum of 80% coverage rather than allowing such a large drop?

I think so, yes. The naming is a bit confusing.

So maybe threshold: 20% or target: 80%?

[felixarntz]

@swissspidy Since you raised some valid points in #1792 (comment), it would be great to get your feedback whether this solution would address them.

[swissspidy]

@ShyamGadde Appreciate the thorough comparison here 💪 👍

While individual flags per plugin are tempting, I think we can start with the simpler option 2 for now and then see how well it works.

[ShyamGadde]

@westonruter Would you prefer that we display the carried forward flags in the PR comment, as shown below, or would you rather exclude them?

Ref: https://docs.codecov.com/docs/carryforward-flags#advanced-configuring-carryforward-flags-in-the-code-host

[westonruter]

I suppose it doesn't hurt to include?

[swissspidy]

@phanduynam No it's not, please stop spamming the repo with these comments.

[thelovekesh]

This action was recently compromised. My recommendation would be to pin it to a specific SHA for better security.

Additionally, reviewing the workflow, I noticed there's quite a bit of scripting involved, which might not be the easiest to follow for all developers.

To improve clarity and flexibility, I'd suggest using a JavaScript or PHP script instead. This approach would make the code more accessible to a wider range of developers and allow for easier management of edge cases.

You can find a similar composite action here - https://github.com/ampproject/amp-wp/tree/develop/.github/actions/determine-changed-files

[westonruter]

What if we pin it to a vulnerable version?

[westonruter]

But it seems like based on the latest supply chain attack with qix today, pinning is a way to avoid being exploited. Perhaps Dependabot includes some checks for the dependencies it is proposing updates for which would address the concern of pinning to a vulnerable version? It doesn't seem this is the case right now, but this would sure be a great use of AI.

[thelovekesh]

The best way to avoid these issues is to use your own solution rather than relying on third-party actions. It's generally a good idea to either use @actions/* or create your own composite actions.

Also, in core, it's very rare for maintainers to allow third-party actions, and when they do, they always pin to a specific SHA to mitigate potential security risks.

What if we pin it to a vulnerable version?

By default, pin to the SHA of a version that was published at least a week ago. This provides ample time to detect if the action has been compromised.

[westonruter]

Thank you for the advice. I've opened a PR to implement this: #2171

[westonruter]

I just saw a mention of this action being compromised back in March: https://semgrep.dev/blog/2025/popular-github-action-tj-actionschanged-files-is-compromised/

This was around when the commits were being pushed to this branch. I suppose this means we should rotate the secrets? Granted, it has been 9 months.

[westonruter]

Suggested change

	uses: tj-actions/changed-files@v45
	uses: tj-actions/changed-files@e0021407031f5be11a464abee9a0776171c79891 # v47.0.1

[adamsilverstein]

the indenting on this line doesn't match the other plugins

[westonruter]

Yeah. There is also a merge conflict due to #1605. So that merge conflict needs to be fixed, and then npm run format-js should fix up this YAML file as well.

[ShyamGadde]

Fixed it in bc5cfe9.

[Copilot]

Pull request overview

This PR optimizes CI/CD by running unit tests only for plugins affected by changes in pull requests, while maintaining full test coverage for commits to the trunk branch. It introduces Codecov flag management to track coverage per plugin and implements plugin dependency detection.

Changes:

• Added Codecov flag management to enable per-plugin coverage tracking with carryforward support
• Modified the test workflow to detect changed plugins and run tests selectively
• Implemented dependency-aware testing (e.g., changes to optimization-detective trigger tests for embed-optimizer and image-prioritizer)

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File	Description
codecov.yml	Configured per-plugin coverage flags, carryforward rules, and adjusted coverage thresholds for selective testing
.github/workflows/php-test-plugins.yml	Added changed file detection, plugin dependency logic, selective test execution, and direct Codecov CLI integration

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The patch threshold was lowered from 80% to 20%, which is a significant decrease that could allow code with poor test coverage to be merged. This change should be documented in the PR description or reconsidered, as it represents a substantial reduction in code quality standards.

Suggested change

	threshold: 20%
	threshold: 80%

[Copilot]

The shell script doesn't handle the case where no plugins have changed. If ALL_CHANGED_PLUGINS is empty, the subsequent for loops in the test execution steps will fail silently or produce errors. Add a check to ensure at least one plugin needs testing, or exit early with a message if no plugins changed.

Suggested change

	if [[ ${#ALL_CHANGED_PLUGINS[@]} -eq 0 ]]; then
	echo "No changed plugins detected; skipping plugin tests."
	echo "all_changed_plugins=" >> $GITHUB_OUTPUT
	exit 0
	fi

[Copilot]

The codecov CLI verification steps create a .gpg directory but don't clean it up afterward. This could leave sensitive keyring files on the runner. Add a cleanup step or use the system's default GPG directory instead.

[Copilot]

The codecov upload command assumes that coverage files exist for all changed plugins, but if a test execution fails for any plugin, the corresponding coverage file won't be generated. This will cause the upload to fail with --fail-on-error. Consider checking if files exist before attempting to upload them, or handle missing files gracefully.

Suggested change

	./codecov upload-coverage "${cc_args[@]}" \
	--file ./single-site-reports/coverage-$PLUGIN.xml \
	--file ./multisite-reports/coverage-multisite-$PLUGIN.xml \
	--flag $PLUGIN \
	--name $PLUGIN-coverage
	single_site_report="./single-site-reports/coverage-$PLUGIN.xml"
	multisite_report="./multisite-reports/coverage-multisite-$PLUGIN.xml"
	files=()
	if [ -f "$single_site_report" ]; then
	files+=(--file "$single_site_report")
	else
	echo "::warning::Single-site coverage report not found for plugin '$PLUGIN' at '$single_site_report'."
	fi
	if [ -f "$multisite_report" ]; then
	files+=(--file "$multisite_report")
	else
	echo "::warning::Multisite coverage report not found for plugin '$PLUGIN' at '$multisite_report'."
	fi
	if [ ${#files[@]} -eq 0 ]; then
	echo "::warning::No coverage reports found for plugin '$PLUGIN'. Skipping Codecov upload for this plugin."
	echo "::endgroup::"
	continue
	fi
	./codecov upload-coverage "${cc_args[@]}" "${files[@]}" \
	--flag "$PLUGIN" \
	--name "$PLUGIN-coverage"

[Copilot]

The script uses unquoted variable expansion in the for loops (${{ steps.changed-plugins.outputs.all_changed_plugins }}), which could cause issues if plugin names contain spaces or special characters. While plugin names in this repository don't have spaces, it's a best practice to quote these expansions for robustness.

[Copilot]

The condition checks if github.event_name equals "push" but doesn't verify which branch was pushed to. According to the workflow trigger configuration, pushes can occur on both "trunk" and "release/**" branches. The description states "All tests will run for all plugins when commits are added to the trunk branch", but this condition will run all tests for any push, including release branches. Consider adding a branch check to ensure this behavior only applies to trunk pushes if that's the intended behavior.