Modernize wads

[thorwhalen]

Modernize wads: adopt pyproject-based workflows, modern linters/formatters, reproducible templates, and modern GitHub Actions for packaging & publishing

Summary
wads already provides two very useful high-level features for Python projects:

• populate: quickly scaffold a parametrized project (folders, files, basic config).
• pack: synchronize metadata, bump versions, and publish/update a package on PyPI.

This issue proposes modernizing wads so it integrates first-class with the modern Python ecosystem (pyproject.toml, PEP 621, build backends) and modern developer tools (ruff, black, isort, pre-commit, pytest, hatch/poetry/flit), and uses up-to-date GitHub features (Actions reusable workflows, Templates, OIDC publishing where possible). The goal is to make generated projects and the pack flow safer, more reproducible, and friendlier to contributors.

Why now

• The Python ecosystem has coalesced around pyproject.toml and modern tooling, which improves interoperability and decreases boilerplate.
• Linters/formatters like ruff and black dramatically reduce PR friction and CI flakiness.
• GitHub has better features for templates, reusable Actions, and secure publishing (OIDC) that reduce secret management and maintenance cost.
• Users expect easy, modern defaults from a project scaffolder / pack tool.

Proposed changes (what to add to wads and generated projects)

1. Adopt pyproject.toml by default

   • Generate pyproject.toml for new projects (PEP 621 metadata where appropriate) instead of setup.py + setup.cfg glue.
   • Support popular build backends like Hatchling, Flit, or Setuptools (with PEP 517/518).
   • Links:
     • https://peps.python.org/pep-0621/
     • https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

2. Make populate produce modern, linted, and testable projects

   • Include minimal sensible config files:
     • pyproject.toml (project metadata, build-backend, tool configs)
     • .editorconfig
     • .gitignore, LICENSE and README.md templates
     • GitHub repository template files (issue/PR templates, CODEOWNERS optional)
   • Provide optional integration toggles (via parameters) for:
     • Linter: ruff (https://github.com/charliermarsh/ruff)
     • Formatter: black (https://github.com/psf/black)
     • Import sorter: isort (https://pycqa.github.io/isort/)
     • Type checker: mypy (https://github.com/python/mypy) or pyright
     • Test runner: pytest (https://github.com/pytest-dev/pytest)
     • Test orchestration: nox or tox where needed
   • Configure these tools via pyproject.toml so there is a single source of truth. Example (conceptually):
     • tool.ruff, tool.black, tool.isort, tool.pytest, tool.mypy in pyproject.toml

3. Use ruff as the primary linter and formatter early

   • Ruff is extremely fast, can replace many of flake8/pyflakes/isort runs, and includes autofix capability.
   • Use black for formatting where desired, or enable ruff's formatting rules if using ruff for everything.
   • Provide a recommended pre-commit configuration (https://pre-commit.com/) to run linters/formatters locally and in CI.

4. Modern packaging & publish: support build + safe publish

   • Implement pack to:
     • Ensure pyproject metadata and version are in sync (canonical single source)
     • Build sdist/wheel using python -m build (https://pypa-build.readthedocs.io/)
     • Validate packages (twine check) and optionally run auditwheel where relevant
     • Publish to PyPI via twine upload or GitHub Actions (recommended)
   • Recommend backends: Hatchling (https://hatch.pypa.io/), Flit (https://flit.readthedocs.io/), or modern setuptools with setup.cfg as needed.
   • Support version bump strategies: direct (bump in code/pyproject) or use canonical version file. Consider supporting semantic-release approaches.

5. GitHub Actions: reusable workflows & secure publishing

   • Provide a set of reusable GitHub Actions workflows in the template repo:
     • CI workflow: checkout -> setup-python -> install deps -> ruff/black/pytest -> build-> publish on release
       • Actions to consider: actions/checkout, actions/setup-python, EndBug/add-and-commit (for automated version commits), pypa/gh-action-pypi-publish
     • Use reusable/workflow_call patterns so downstream repos can call common workflows: https://docs.github.com/en/actions/using-workflows/reusing-workflows
     • Use OIDC where possible to avoid long-lived PyPI tokens in repository secrets (GitHub OIDC + ephemeral tokens / pypi support) — if not possible, document best practices for storing PyPI API tokens as repository/organization secrets.
     • Add Dependabot config to keep dependencies and Actions up to date.

6. GitHub repository Templates & starter workflow

   • Make wads provide a GitHub "template repository" that can be used to create new repositories with the recommended structure and GitHub-specific files (issue templates, community health files).
   • Add optional GitHub Templates support (https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-template-repository) so populate can also generate from a maintained template.

7. Developer experience improvements

   • Interactive or CLI flags for populate to choose which optional features to enable (CI, linting, test, type-checking, packaging backend).
   • Offer a --minimal mode for simple projects and an --opinionated mode for full modern defaults.
   • Add wads doctor or wads check to validate project layout, pyproject metadata, CI workflow syntax and that pack would succeed locally.

How these changes help (benefits)

• Better interoperability: pyproject.toml standardizes tool configuration and packaging metadata so downstream tools work consistently.
• Faster developer feedback: ruff + pre-commit + Actions yields quick, consistent linting and formatting across machines and CI.
• Safer publishing: using build + twine and modern Actions patterns reduces accidental bad releases; OIDC reduces secret sprawl.
• Lower maintenance: reusable GitHub Actions and template repos centralize updates; Dependabot keeps deps current.
• Easier onboarding: generated projects have a consistent layout and CI, lowering cognitive overhead for contributors.
• Future-proofing: aligning with PEPs and modern build backends prevents technical debt and reduces migration pain later.

Recommended concrete defaults (suggested)

• Project metadata: pyproject.toml (PEP 621)
• Build backend: Hatchling (default) or Flit (opt-in)
• Lint: ruff (with autofix enabled as an option)
• Format: black (or ruff formatting)
• Imports: isort (configured within pyproject)
• Tests: pytest
• Pre-commit: include hooks for ruff/black/isort
• CI: reusable GitHub Actions workflows with:
  • Workflow for PRs: lint + tests
  • Workflow for releases: build + twine publish (triggered by tag/release)
• Publishing: prefer GitHub Actions + OIDC if feasible; otherwise use PyPI token stored in repository secrets and document rotation.

Migration plan (high level)

1. Add pyproject.toml support to wads and add options for choosing a build backend.
2. Update populate templates to include the recommended tool configs and an initial GitHub Actions workflow (as reusable templates).
3. Update pack to:
   • Read metadata from pyproject.toml
   • Build via python -m build
   • Optionally run validation steps (twine check)
   • Emit artifacts or call a publish action
4. Publish a v2.0 release of wads documenting breaking changes and migration steps for existing users (guide to convert setup.py -> pyproject).
5. Provide example repositories / template repository showcasing full modern stack.

Acceptance criteria

• Generated projects use pyproject.toml and build successfully with the chosen backend.
• populate provides options to enable ruff/black/isort/pre-commit and the resulting project passes those checks locally.
• pack builds sdist + wheel and (optionally) publishes to a test PyPI repository in CI.
• Reusable GitHub Actions are available and documented, and a template repository exists for users to adopt.

Risks & mitigation

• Backwards compatibility: existing users may still rely on setup.py workflows. Mitigate by supporting an opt-in migration flow and clear migration docs.
• Tool churn: choose sensible defaults (ruff/black/pytest/hatchling) and make them configurable.
• Security of publishing: document OIDC flow and fallback to rotated PyPI tokens for orgs that prefer tokens.

References & resources

• pyproject/PEP 621: https://peps.python.org/pep-0621/
• Ruff: https://github.com/charliermarsh/ruff
• Black: https://github.com/psf/black
• isort: https://pycqa.github.io/isort/
• pytest: https://github.com/pytest-dev/pytest
• Hatchling: https://hatch.pypa.io/
• Flit: https://flit.readthedocs.io/
• Python build: https://pypa-build.readthedocs.io/
• Twine: https://github.com/pypa/twine
• GitHub Actions reusable workflows: https://docs.github.com/en/actions/using-workflows/reusing-workflows
• GitHub template repositories: https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-template-repository
• GitHub OIDC: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect

Remember to:

• look at, and take into account wads existing issues
• coordinate with the isee (CI) project, whose defaults depend on wads defaults

Next steps

• Decide the scope (minimal compatibility vs opinionated v2) and which defaults to adopt.
• Prepare a branch that:
  • Adds pyproject templates and tool configs to populate.
  • Implements pyproject-aware pack (build -> artifacts -> publish).
  • Adds a template repo or folder with reusable GitHub Actions.
• Document migration steps and release notes.

Please let me know which defaults you want enforced (e.g., Hatchling vs Flit vs Poetry, ruff-only vs ruff+black), and I will open a PR that implements the first-pass changes (populate templates + a pyproject-aware pack flow).