CI: Test Windows / Visual Studio 2022, and add automatic dependent repo resolution logic.

[biergaizi]

Introduction

This Pull Request introduces two major new features and performs some housekeeping tasks to the CI/CD pipeline:

1. Add automatic dependent-repo resolution logic.

2. Test Windows / Visual Studio 2022 using MSVC and clang-cl.

   • vcpkg manifest files for automatic dependency installation.
   • cache compiled vcpkg packages using GitHub Actions cache.

3. Miscellaneous housekeeping changes.

Not implemented:

1. Unit tests and smoke tests on Windows.
2. Python extensions on Windows.

To be implement in the future.

CI: Add automatic dependent repo resolution logic.

The openEMS Project is a multi-repo project. Changes frequently affect multiple repos, with changes depending on each other. Forking a repo also means all related repos must be forked together, creating a poor contribution experience, since the CI pipeline doesn't work properly for them.

This commit introduces automatic dependent-repo resolution logic to the CI/CD pipeline, with the following rules.

1. Want to develop a multi-repo change in CSXCAD and openEMS? Use the same branch name, they'll be tested together.

2. Want to send Pull Requests to both CSXCAD and openEMS? Use the same branch name in your fork, and open two individual Pull Requests against CSXCAD and openEMS. If both Pull Requests are opened, and the source branch name is the same, they'll be tested together.

3. Want to fork openEMS and develop it at the downstream? You can fork the only the repos you need. The CI/CD script automatically uses the project founder's repos as fall back for non-forked dependencies. You can override these dependencies by forking more repos.

Owner-then-Founder Dependency Lookup

If a git commit is pushed into a repo, or if a repo has received a Pull Request from a contributor, to build this repo, we need to look for the repo's dependencies. We check all dependencies one by one. If the repo's owner also has dependent repo under their GitHub account. If it's the case, the owner's copy is used as the dependency. If it's not the case, the founder account thliebig's copy is used as the dependency.

This two-tier lookup solves several problems:

1. An experimenter can only fork openEMS without forking any dependencies. Since the CI/CD script falls back to the founder's repos for dependencies, their local repo's CI/CD works automatically. If they want to make a change to CSXCAD and openEMS at the same time, they only need to selectively fork CSXCAD, so that the "repo owner override" takes effect. At the same time, they can keep using the founder's fparser repo as a fallback without forking it.

2. Someone may send a Pull Request against a downstream fork itself, rather than the project founder. For example, I have a downstream openEMS fork named fasterEMS. If someone sends a Pull Request against my "fasterEMS/CSXCAD", my repo's CI/CD needs to use the same-owner repo "fasterEMS/fparser" as the dependency, not project founder's "thliebig/fparser".

Pull-Request Ganging

If a Pull Request "pr_primary" is submitted against the upstream repo "repo_primary", we check whether the same contributor has also opened another Pull Request "pr_dependency" against our dependency "repo_dependency", and whether both Pull Requests uses the same source branch name, such as "feature". If both conditions are met, we can say that two Pull Requests are "ganged", they are tested by CI/CD together. When testing "pr_primary" at "repo_primary", instead of using default repos as dependencies, we checkout the merge commit of "pr_secondary" at the "repo_secondary" as its dependency.

For more than 2 repos, the same "ganged PR" logic also applies.

If Pull-Request Ganging is activated, a warning is generated in the "Annotations" panel of the GitHub Actions Summary page, reminding developers to merge the dependent PR first before the main PR.

Branch Ganging

If a git commit is submitted to a non-default branch "feature" of repo "repo_primary", we check whether our dependency "repo_dependency" also has a branch named "feature".

If this condition is met, we can say that two repo branches are "ganged", they are tested by CI/CD together. When testing the branch "feature" of "repo_primary", instead of using default branch "master" of "repo_dependency" as the dependency, we checkout the branch "feature" the "repo_dependency" instead.

This allows testing multi-repo features together while it's in development. For more than 2 repos, the same "ganged branches" logic also applies.

If Branch Ganging is activated, a warning is generated in the "Annotations" panel of the GitHub Actions Summary page, reminding developers that a different branch of the dependent repo is used for this test.

Implementation Details

This commit adds a new CI job called "Resolve Repo Dependencies", which is executed before all other jobs. This job calls the Python script "scripts/resolve_dependent_repos.py", which uses GitHub Actions variables and GitHub APIs to find the needed dependencies according to the conditions and contexts explained above.

After the script finishes, it generates several output variables that contain the repoes and branches of all dependencies. These output variables are passed using the shell variable "$GITHUB_OUTPUT" (provided by GitHub Actions), later, all other jobs use these variable as their inputs for the "actions/checkout" steps.

vcpkg

vcpkg: add manifest for managing Windows dependencies.

This commit introduces a project manifest for managing Windows dependencies using vcpkg, the modern C/C++ package manager for Windows development.

Although vcpkg is a Microsoft product, it's a free and open source package manager. In principle, vcpkg can also manage dependencies and even make binary releases on Unix-like systems, but this use case is untested and unsupported. Use at one's own risks. Currently, the use case is strictly for auto-building dependencies on Windows.

Note: since we have to maintain the vcpkg pipeline anyway, perhaps it does make sense to reuse vcpkg for releasing binary packages for both Windows and Unix-like systems in the future, but this remains a speculative idea for now.

CI: Windows MSVC/clang-cl Test using Visual Studio 2022.

This commit introduces Continuous Integration for Windows with MSVC/clang-cl shipped in Visual Studio 2022.

vcpkg is used to automatically build all dependencies from source, with the advantage of full automation, latest upstream version, perfect binary compatibility, and customizable build flags.

However, the full source build takes a long time (like using Gentoo Portage or MacPorts), in this case, 2 hours. A vcpkg binary cache is used to speedup iterative development. GitHub Actions cache is immutable, we can't modify existing cache, so we use a last-recently logic: Before each run, the last-written cache with a given prefix is restored. After each run, a new cache with the same prefix and a different suffix is submitted to GitHub. It's GitHub's responsibility to delete old cache if 7 days have passed or after the total size exceeds 10 GiB.

Housekeeping

CI: use GitHub Actions env block, not bash_profile.

For the Linux matrix, use GitHub Actions native "env" block to define CXXFLAGS, not bash_profile.

CI: Don't install python3 and cmake on macOS.

Homebrew always warns that python3 and cmake are already installed, this causes warning fatigue in the GitHub Actions Summary's "Annotations" panel, developers may miss other useful warnings.

CMakeLists.txt: quote VTK_VERSION and CGAL_VERSION

If VTK_VERSION is an empty string (because VTK is not installed), the statement if (${VTK_VERSION} VERSION_GREATER "9") is evaluated as if (VERSION_GREATER "9"), generating the following misleading and confusing syntax error:

    CMake Error at CMakeLists.txt:151 (if):
      if given arguments:

        "VERSION_GREATER" "9"

      Unknown arguments specified

Quote VTK_VERSION to avoid the error. We do the same for CGAL_VERSION for consistency (although CGAL_VERSION shouldn't be affected by the same problem since errors are not ignored)

[biergaizi]

Merge ready.

Some remarks.

• No Python on Windows yet, to be implemented in a future version.
• You can see a real-world example of the "Pull-Request Ganging" feature at https://github.com/thliebig/openEMS/actions/runs/21083710338?pr=208. Note that the openEMS update depends on the CSXCAD update, so the openEMS CI pipeline automatically finds and uses this CSXCAD pull request as its dependency.