From 025833a182afd664edc391ed57e02fae6bb3aee5 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 24 Oct 2025 22:13:53 +0000 Subject: [PATCH 1/5] Initial plan From 239db6627c4032c3354e6f72bbe85fad161d89e8 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 24 Oct 2025 22:19:51 +0000 Subject: [PATCH 2/5] Make pyproject.toml single source of truth for dependencies - Migrate all dependencies to pyproject.toml with version pinning - Update setup.py to read from pyproject.toml - Create script to generate requirements files - Auto-generate requirements.txt, requirements_dev.txt, environment.yml, environment_dev.yml - Add comprehensive dependency management documentation - Update CONTRIBUTING.rst with new dependency workflow Co-authored-by: ladc <1212574+ladc@users.noreply.github.com> --- CONTRIBUTING.rst | 26 ++- .../developer_guide/dependency_management.rst | 189 ++++++++++++++++ environment.yml | 27 +-- environment_dev.yml | 57 +++-- pyproject.toml | 120 ++++++++++- requirements.txt | 22 +- requirements_dev.txt | 51 +++-- scripts/README.md | 30 +++ scripts/generate_requirements.py | 202 ++++++++++++++++++ setup.py | 49 +---- 10 files changed, 656 insertions(+), 117 deletions(-) create mode 100644 doc/source/developer_guide/dependency_management.rst create mode 100644 scripts/README.md create mode 100755 scripts/generate_requirements.py diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 473297b14..88895d7b0 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -362,6 +362,23 @@ This will help to reduce the time needed to process the PR. For changes outside recommend opening a new PR. +Managing dependencies +~~~~~~~~~~~~~~~~~~~~~ + +Pysteps uses **pyproject.toml** as the single source of truth for all dependencies. +The files ``requirements.txt``, ``requirements_dev.txt``, ``environment.yml``, and +``environment_dev.yml`` are auto-generated and should **NOT** be edited manually. + +To update dependencies: + +1. Edit the appropriate section in ``pyproject.toml`` +2. Run ``python scripts/generate_requirements.py`` to regenerate all requirement files +3. Commit both ``pyproject.toml`` and the generated files + +For more information, see the +`Dependency Management Guide `__. + + Testing your changes ~~~~~~~~~~~~~~~~~~~~ @@ -437,12 +454,13 @@ Core developers should follow the steps to prepare a new release (version): 1. Before creating the actual release in GitHub, be sure that every item in the following checklist was followed: - * In the file setup.py, update the **version="X.X.X"** keyword in the setup - function. + * In the file pyproject.toml, update the **version="X.X.X"** in the + [project] section. * Update the version in PKG-INFO file. * If new dependencies were added to pysteps since the last release, add - them to the **environment.yml, requirements.txt**, and - **requirements_dev.txt** files. + them to **pyproject.toml** in the appropriate section ([project.dependencies] + or [project.optional-dependencies]), then regenerate the requirements files by + running ``python scripts/generate_requirements.py``. #. Create a new release in GitHub following `these guidelines `_. Include a detailed changelog in the release. diff --git a/doc/source/developer_guide/dependency_management.rst b/doc/source/developer_guide/dependency_management.rst new file mode 100644 index 000000000..8061b2edd --- /dev/null +++ b/doc/source/developer_guide/dependency_management.rst @@ -0,0 +1,189 @@ +Dependency Management +===================== + +Overview +-------- + +As of version 1.18.1, pysteps uses **pyproject.toml** as the single source of truth +for all project dependencies following `PEP 621 `_ +standards. This ensures reproducible builds and prevents unpredictable failures due to +dependency upgrades. + +All version constraints are explicitly defined to balance stability and compatibility: + +- **Minimum versions** ensure required features are available +- **Maximum versions** prevent breaking changes from future releases +- All dependencies are pinned with version ranges (e.g., ``>=1.24.0,<3.0``) + +Source of Truth: pyproject.toml +-------------------------------- + +The ``pyproject.toml`` file contains: + +1. **Core dependencies** (``[project.dependencies]``) + + Required packages for basic pysteps functionality: + + - numpy + - opencv-python + - pillow + - pyproj + - scipy + - matplotlib + - jsmin + - jsonschema + - netCDF4 + +2. **Optional dependencies** (``[project.optional-dependencies]``) + + Organized by feature group: + + - ``performance``: dask, pyfftw + - ``geo``: cartopy, rasterio + - ``io``: h5py, pygrib + - ``analysis``: scikit-image, scikit-learn, pandas, PyWavelets + - ``all``: All optional dependencies combined + - ``dev``: Development tools and all optional dependencies + - ``docs``: Documentation building requirements + +Generated Files +--------------- + +The following files are **auto-generated** from ``pyproject.toml`` and should +**NOT be edited manually**: + +- ``requirements.txt`` - pip requirements for core dependencies +- ``requirements_dev.txt`` - pip requirements including dev dependencies +- ``environment.yml`` - conda environment for core dependencies +- ``environment_dev.yml`` - conda environment including dev dependencies + +These files are maintained for backwards compatibility and convenience. + +Updating Dependencies +--------------------- + +To update dependencies in the project: + +1. Edit the appropriate section in ``pyproject.toml`` +2. Run the generation script:: + + python scripts/generate_requirements.py + +3. The script will regenerate all requirements files automatically + +Example: Adding a New Dependency +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To add a new core dependency: + +1. Edit ``pyproject.toml``:: + + [project] + dependencies = [ + # ... existing dependencies ... + "new-package>=1.0.0,<2.0", + ] + +2. Regenerate files:: + + python scripts/generate_requirements.py + +3. Commit both ``pyproject.toml`` and the generated files + +Installing pysteps +------------------ + +With pip +~~~~~~~~ + +Install core dependencies:: + + pip install . + +Install with optional features:: + + pip install .[all] # All optional dependencies + pip install .[geo] # Only geospatial features + pip install .[performance] # Only performance enhancements + +For development:: + + pip install -e .[dev] + +With conda +~~~~~~~~~~ + +Using the generated environment file:: + + conda env create -f environment.yml + conda activate pysteps + +For development:: + + conda env create -f environment_dev.yml + conda activate pysteps_dev + +Dependency Version Policy +-------------------------- + +Version constraints follow these principles: + +1. **Minimum version**: Set to a version known to work, typically from when + the dependency was added or last tested + +2. **Maximum version**: Set to the next major version to prevent breaking changes + Example: If minimum is 1.24.0, maximum is typically <3.0 or <2.0 + +3. **Python compatibility**: Tested on Python 3.11, 3.12, and 3.13 + +4. **Regular updates**: Dependencies should be reviewed and updated periodically + to benefit from bug fixes and new features while maintaining stability + +Automated Dependency Updates +----------------------------- + +Consider using dependabot or similar tools to: + +- Monitor for security vulnerabilities +- Suggest updates to dependency versions +- Test compatibility with newer versions + +To enable dependabot, add a ``.github/dependabot.yml`` configuration file. + +Troubleshooting +--------------- + +If you encounter dependency conflicts: + +1. Verify you're using a supported Python version (3.11-3.13) +2. Try creating a fresh virtual environment +3. Check if mixing pip and conda installations caused conflicts +4. Review the dependency versions in ``pyproject.toml`` + +Common Issues +~~~~~~~~~~~~~ + +**"No matching distribution found"** + One of the dependencies may not be available for your platform. + Check the package's PyPI page for platform availability. + +**Version conflicts** + If you see version conflicts, ensure you're not mixing packages from + different sources (pip vs conda). Use one package manager consistently. + +**Build failures** + Some packages like pygrib may require system libraries. Install + required system dependencies using your system package manager. + +Migration from Old System +-------------------------- + +Prior to version 1.18.1, dependencies were scattered across multiple files. +The migration involved: + +1. Consolidating all dependencies into ``pyproject.toml`` +2. Adding explicit version constraints +3. Creating generation scripts for backwards compatibility +4. Updating ``setup.py`` to use ``pyproject.toml`` + +Old dependency files are now auto-generated and should not be edited manually. diff --git a/environment.yml b/environment.yml index bfa7bfdb8..132687564 100644 --- a/environment.yml +++ b/environment.yml @@ -1,15 +1,18 @@ +# This file is auto-generated from pyproject.toml +# To regenerate, run: python scripts/generate_requirements.py +# DO NOT EDIT THIS FILE MANUALLY name: pysteps channels: -- conda-forge -- defaults + - conda-forge + - defaults dependencies: - - python>=3.10 - - jsmin - - jsonschema - - matplotlib - - netCDF4 - - numpy - - opencv - - pillow - - pyproj - - scipy + - python>=3.11 + - numpy>=1.24.0,<3.0 + - opencv>=4.7.0,<5.0 + - pillow>=10.0.0,<12.0 + - pyproj>=3.5.0,<4.0 + - scipy>=1.10.0,<2.0 + - matplotlib>=3.7.0,<4.0 + - jsmin>=3.0.0,<4.0 + - jsonschema>=4.0.0,<5.0 + - netCDF4>=1.6.0,<2.0 diff --git a/environment_dev.yml b/environment_dev.yml index 7ea35ad4f..7f9286452 100644 --- a/environment_dev.yml +++ b/environment_dev.yml @@ -1,34 +1,33 @@ -# pysteps development environment +# This file is auto-generated from pyproject.toml +# To regenerate, run: python scripts/generate_requirements.py +# DO NOT EDIT THIS FILE MANUALLY name: pysteps_dev channels: - conda-forge - defaults dependencies: - - python>=3.10 - - pip - - jsmin - - jsonschema - - matplotlib - - netCDF4 - - numpy - - opencv - - pillow - - pyproj - - scipy - - pytest - - pywavelets - - cython - - dask - - pyfftw - - h5py - - PyWavelets - - pygrib - - black - - pytest-cov - - codecov - - pre_commit - - cartopy>=0.18 - - scikit-image - - scikit-learn - - pandas - - rasterio + - python>=3.11 + - numpy>=1.24.0,<3.0 + - opencv>=4.7.0,<5.0 + - pillow>=10.0.0,<12.0 + - pyproj>=3.5.0,<4.0 + - scipy>=1.10.0,<2.0 + - matplotlib>=3.7.0,<4.0 + - jsmin>=3.0.0,<4.0 + - jsonschema>=4.0.0,<5.0 + - netCDF4>=1.6.0,<2.0 + - pytest>=7.3.0,<9.0 + - pytest-cov>=4.1.0,<7.0 + - black>=23.3.0,<25.0 + - pre-commit>=3.3.0,<5.0 + - Cython>=0.29.2,<4.0 + - dask>=2023.5.0,<2025.0 + - pyfftw>=0.13.0,<1.0 + - cartopy>=0.22.0,<1.0 + - rasterio>=1.3.0,<2.0 + - h5py>=3.8.0,<4.0 + - pygrib>=2.1.0,<3.0 + - scikit-image>=0.20.0,<1.0 + - scikit-learn>=1.3.0,<2.0 + - pandas>=2.0.0,<3.0 + - pywavelets>=1.4.0,<2.0 diff --git a/pyproject.toml b/pyproject.toml index 17b9b0036..5ec8e8a5d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -10,11 +10,129 @@ requires = [ build-backend = "setuptools.build_meta:__legacy__" #https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support +[project] +name = "pysteps" +version = "1.18.1" +description = "Python framework for short-term ensemble prediction systems" +readme = "README.rst" +license = {text = "BSD-3-Clause"} +authors = [ + {name = "PySteps developers"} +] +requires-python = ">=3.11,<3.14" +classifiers = [ + "Development Status :: 5 - Production/Stable", + "Intended Audience :: Science/Research", + "Topic :: Scientific/Engineering", + "Topic :: Scientific/Engineering :: Atmospheric Science", + "Topic :: Scientific/Engineering :: Hydrology", + "License :: OSI Approved :: BSD License", + "Programming Language :: Python :: 3 :: Only", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Operating System :: OS Independent", +] + +# Core runtime dependencies (pinned for reproducibility) +dependencies = [ + "numpy>=1.24.0,<3.0", + "opencv-python>=4.7.0,<5.0", + "pillow>=10.0.0,<12.0", + "pyproj>=3.5.0,<4.0", + "scipy>=1.10.0,<2.0", + "matplotlib>=3.7.0,<4.0", + "jsmin>=3.0.0,<4.0", + "jsonschema>=4.0.0,<5.0", + "netCDF4>=1.6.0,<2.0", +] + +[project.optional-dependencies] +# Performance enhancements +performance = [ + "dask>=2023.5.0,<2025.0", + "pyfftw>=0.13.0,<1.0", +] + +# Visualization and geospatial +geo = [ + "cartopy>=0.22.0,<1.0", + "rasterio>=1.3.0,<2.0", +] + +# Data I/O +io = [ + "h5py>=3.8.0,<4.0", + "pygrib>=2.1.0,<3.0; sys_platform != 'win32'", +] + +# Advanced analysis +analysis = [ + "scikit-image>=0.20.0,<1.0", + "scikit-learn>=1.3.0,<2.0", + "pandas>=2.0.0,<3.0", + "PyWavelets>=1.4.0,<2.0", +] + +# All optional dependencies +all = [ + "dask>=2023.5.0,<2025.0", + "pyfftw>=0.13.0,<1.0", + "cartopy>=0.22.0,<1.0", + "rasterio>=1.3.0,<2.0", + "h5py>=3.8.0,<4.0", + "pygrib>=2.1.0,<3.0; sys_platform != 'win32'", + "scikit-image>=0.20.0,<1.0", + "scikit-learn>=1.3.0,<2.0", + "pandas>=2.0.0,<3.0", + "PyWavelets>=1.4.0,<2.0", +] + +# Development dependencies +dev = [ + "pytest>=7.3.0,<9.0", + "pytest-cov>=4.1.0,<7.0", + "black>=23.3.0,<25.0", + "pre-commit>=3.3.0,<5.0", + "Cython>=0.29.2,<4.0", + # Include all optional dependencies for testing + "dask>=2023.5.0,<2025.0", + "pyfftw>=0.13.0,<1.0", + "cartopy>=0.22.0,<1.0", + "rasterio>=1.3.0,<2.0", + "h5py>=3.8.0,<4.0", + "pygrib>=2.1.0,<3.0; sys_platform != 'win32'", + "scikit-image>=0.20.0,<1.0", + "scikit-learn>=1.3.0,<2.0", + "pandas>=2.0.0,<3.0", + "PyWavelets>=1.4.0,<2.0", +] + +# Documentation dependencies +docs = [ + "sphinx>=5.0.0,<9.0", + "sphinxcontrib-bibtex>=2.5.0,<3.0", + "sphinx-book-theme>=1.0.0,<2.0", + "sphinx-gallery>=0.13.0,<1.0", + "scikit-image>=0.20.0,<1.0", + "scikit-learn>=1.3.0,<2.0", + "pandas>=2.0.0,<3.0", + "pygrib>=2.1.0,<3.0; sys_platform != 'win32'", + "h5py>=3.8.0,<4.0", +] + +[project.urls] +Homepage = "https://pysteps.github.io/" +Documentation = "https://pysteps.readthedocs.io" +Repository = "https://github.com/pySTEPS/pysteps" +Issues = "https://github.com/pySTEPS/pysteps/issues" +Changelog = "https://github.com/pySTEPS/pysteps/releases" +CI = "https://github.com/pySTEPS/pysteps/actions" # Define black parameters for the project # https://black.readthedocs.io/en/stable/pyproject_toml.html#configuration-format [tool.black] -target-version = ['py36'] +target-version = ['py311'] line-length = 88 exclude = ''' /( diff --git a/requirements.txt b/requirements.txt index 1804df1d9..6c85ac6fa 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,9 +1,13 @@ -numpy -opencv-python -pillow -pyproj -scipy -matplotlib -jsmin -jsonschema -netCDF4 +# This file is auto-generated from pyproject.toml +# To regenerate, run: python scripts/generate_requirements.py +# DO NOT EDIT THIS FILE MANUALLY + +numpy>=1.24.0,<3.0 +opencv-python>=4.7.0,<5.0 +pillow>=10.0.0,<12.0 +pyproj>=3.5.0,<4.0 +scipy>=1.10.0,<2.0 +matplotlib>=3.7.0,<4.0 +jsmin>=3.0.0,<4.0 +jsonschema>=4.0.0,<5.0 +netCDF4>=1.6.0,<2.0 diff --git a/requirements_dev.txt b/requirements_dev.txt index c49ea2d18..5c5dbedff 100644 --- a/requirements_dev.txt +++ b/requirements_dev.txt @@ -1,25 +1,30 @@ -# Base dependencies -python>=3.10 -numpy -opencv-python -pillow -pyproj -scipy -matplotlib -jsmin -jsonschema -netCDF4 +# This file is auto-generated from pyproject.toml +# To regenerate, run: python scripts/generate_requirements.py +# DO NOT EDIT THIS FILE MANUALLY -# Optional dependencies -dask -pyfftw -cartopy>=0.18 -h5py -scikit-image -scikit-learn -pandas -rasterio - -# Testing -pytest +numpy>=1.24.0,<3.0 +opencv-python>=4.7.0,<5.0 +pillow>=10.0.0,<12.0 +pyproj>=3.5.0,<4.0 +scipy>=1.10.0,<2.0 +matplotlib>=3.7.0,<4.0 +jsmin>=3.0.0,<4.0 +jsonschema>=4.0.0,<5.0 +netCDF4>=1.6.0,<2.0 +# Development dependencies +pytest>=7.3.0,<9.0 +pytest-cov>=4.1.0,<7.0 +black>=23.3.0,<25.0 +pre-commit>=3.3.0,<5.0 +Cython>=0.29.2,<4.0 +dask>=2023.5.0,<2025.0 +pyfftw>=0.13.0,<1.0 +cartopy>=0.22.0,<1.0 +rasterio>=1.3.0,<2.0 +h5py>=3.8.0,<4.0 +pygrib>=2.1.0,<3.0; sys_platform != 'win32' +scikit-image>=0.20.0,<1.0 +scikit-learn>=1.3.0,<2.0 +pandas>=2.0.0,<3.0 +PyWavelets>=1.4.0,<2.0 diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 000000000..859ecce61 --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,30 @@ +# Scripts + +This directory contains utility scripts for the pysteps project. + +## generate_requirements.py + +Generates requirements.txt, requirements_dev.txt, environment.yml, and environment_dev.yml +from pyproject.toml (the single source of truth for dependencies). + +**Usage:** + +```bash +python scripts/generate_requirements.py +``` + +**When to run:** + +- After modifying dependencies in `pyproject.toml` +- Before committing dependency changes +- As part of the release preparation process + +**Output:** + +- `requirements.txt` - pip requirements for core dependencies +- `requirements_dev.txt` - pip requirements including dev dependencies +- `environment.yml` - conda environment for core dependencies +- `environment_dev.yml` - conda environment including dev dependencies + +**Note:** The generated files should be committed to the repository for backwards +compatibility and convenience, but should never be edited manually. diff --git a/scripts/generate_requirements.py b/scripts/generate_requirements.py new file mode 100755 index 000000000..8a100df6e --- /dev/null +++ b/scripts/generate_requirements.py @@ -0,0 +1,202 @@ +#!/usr/bin/env python3 +""" +Generate requirements.txt and environment.yml files from pyproject.toml. + +This script reads the dependencies from pyproject.toml (the single source of truth) +and generates the requirements files for backwards compatibility and convenience. + +Usage: + python scripts/generate_requirements.py +""" + +import re +import sys +from pathlib import Path + +try: + import tomllib +except ImportError: + # Python < 3.11 + try: + import tomli as tomllib + except ImportError: + print( + "Error: tomli package required for Python < 3.11\n" + "Install with: pip install tomli" + ) + sys.exit(1) + + +def parse_version_constraint(spec): + """ + Convert PEP 440 version constraints to conda-style constraints. + + Examples: + ">=1.24.0,<3.0" -> ">=1.24.0,<3.0" + ">=4.0.0,<5.0" -> ">=4.0.0,<5.0" + """ + return spec + + +def dependency_to_conda_format(dep): + """ + Convert a PEP 508 dependency string to conda format. + + Handles platform markers and converts package names where needed. + """ + # Remove platform markers for conda (conda handles this differently) + dep = re.sub(r'\s*;\s*.*$', '', dep) + + # Special package name mappings for conda + package_mappings = { + 'opencv-python': 'opencv', + 'PyWavelets': 'pywavelets', + } + + for pip_name, conda_name in package_mappings.items(): + if dep.startswith(pip_name): + dep = dep.replace(pip_name, conda_name, 1) + + return dep + + +def generate_requirements_txt(pyproject_data, output_file, include_dev=False): + """Generate requirements.txt from pyproject.toml.""" + dependencies = pyproject_data.get("project", {}).get("dependencies", []) + + lines = [ + "# This file is auto-generated from pyproject.toml", + "# To regenerate, run: python scripts/generate_requirements.py", + "# DO NOT EDIT THIS FILE MANUALLY", + "", + ] + + # Add core dependencies + for dep in dependencies: + lines.append(dep) + + if include_dev: + lines.append("") + lines.append("# Development dependencies") + dev_deps = ( + pyproject_data.get("project", {}) + .get("optional-dependencies", {}) + .get("dev", []) + ) + for dep in dev_deps: + lines.append(dep) + + lines.append("") # Final newline + + output_file.write_text("\n".join(lines)) + print(f"✓ Generated {output_file}") + + +def generate_environment_yml(pyproject_data, output_file, include_dev=False): + """Generate environment.yml from pyproject.toml.""" + dependencies = pyproject_data.get("project", {}).get("dependencies", []) + python_version = pyproject_data.get("project", {}).get("requires-python", ">=3.11") + + # Extract minimum Python version + python_min = re.search(r'>=(\d+\.\d+)', python_version) + if python_min: + python_spec = f">={python_min.group(1)}" + else: + python_spec = ">=3.11" + + env_name = "pysteps_dev" if include_dev else "pysteps" + + lines = [ + "# This file is auto-generated from pyproject.toml", + "# To regenerate, run: python scripts/generate_requirements.py", + "# DO NOT EDIT THIS FILE MANUALLY", + f"name: {env_name}", + "channels:", + " - conda-forge", + " - defaults", + "dependencies:", + f" - python{python_spec}", + ] + + # Add core dependencies + for dep in dependencies: + conda_dep = dependency_to_conda_format(dep) + lines.append(f" - {conda_dep}") + + if include_dev: + # Add development dependencies + optional_deps = ( + pyproject_data.get("project", {}) + .get("optional-dependencies", {}) + ) + + dev_deps = optional_deps.get("dev", []) + for dep in dev_deps: + # Skip duplicates already in core dependencies + dep_name = re.match(r'^([a-zA-Z0-9_-]+)', dep).group(1) + if not any(d.startswith(dep_name) for d in dependencies): + conda_dep = dependency_to_conda_format(dep) + lines.append(f" - {conda_dep}") + + lines.append("") # Final newline + + output_file.write_text("\n".join(lines)) + print(f"✓ Generated {output_file}") + + +def main(): + """Main function to generate all requirement files.""" + # Get project root directory + script_dir = Path(__file__).parent + project_root = script_dir.parent + + pyproject_file = project_root / "pyproject.toml" + + if not pyproject_file.exists(): + print(f"Error: {pyproject_file} not found") + sys.exit(1) + + # Read pyproject.toml + with open(pyproject_file, "rb") as f: + pyproject_data = tomllib.load(f) + + # Generate requirements files + print("\nGenerating requirement files from pyproject.toml...") + print("=" * 60) + + # requirements.txt + generate_requirements_txt( + pyproject_data, + project_root / "requirements.txt", + include_dev=False + ) + + # requirements_dev.txt + generate_requirements_txt( + pyproject_data, + project_root / "requirements_dev.txt", + include_dev=True + ) + + # environment.yml + generate_environment_yml( + pyproject_data, + project_root / "environment.yml", + include_dev=False + ) + + # environment_dev.yml + generate_environment_yml( + pyproject_data, + project_root / "environment_dev.yml", + include_dev=True + ) + + print("=" * 60) + print("✓ All requirement files generated successfully!") + print("\nNote: These files are auto-generated. To update dependencies,") + print(" edit pyproject.toml and re-run this script.") + + +if __name__ == "__main__": + main() diff --git a/setup.py b/setup.py index 2065c5f86..8d1f8e762 100644 --- a/setup.py +++ b/setup.py @@ -1,4 +1,11 @@ # -*- coding: utf-8 -*- +""" +Setup script for pysteps package. + +Note: All project metadata and dependencies are now defined in pyproject.toml +following PEP 621 standards. This setup.py is maintained primarily for building +Cython extensions. +""" import sys @@ -60,46 +67,10 @@ external_modules = cythonize(extensions, force=True, language_level=3) -requirements = [ - "numpy", - "jsmin", - "scipy", - "matplotlib", - "jsonschema", -] - +# All metadata and dependencies are now defined in pyproject.toml +# This setup() call is minimal and primarily for Cython extension building setup( - name="pysteps", - version="1.18.1", - author="PySteps developers", + ext_modules=external_modules, packages=find_packages(), - license="LICENSE", include_package_data=True, - description="Python framework for short-term ensemble prediction systems", - long_description=open("README.rst").read(), - long_description_content_type="text/x-rst", - url="https://pysteps.github.io/", - project_urls={ - "Source": "https://github.com/pySTEPS/pysteps", - "Issues": "https://github.com/pySTEPS/pysteps/issues", - "CI": "https://github.com/pySTEPS/pysteps/actions", - "Changelog": "https://github.com/pySTEPS/pysteps/releases", - "Documentation": "https://pysteps.readthedocs.io", - }, - classifiers=[ - "Development Status :: 5 - Production/Stable", - "Intended Audience :: Science/Research", - "Topic :: Scientific/Engineering", - "Topic :: Scientific/Engineering :: Atmospheric Science", - "Topic :: Scientific/Engineering :: Hydrology", - "License :: OSI Approved :: BSD License", - "Programming Language :: Python :: 3 :: Only", - "Programming Language :: Python :: 3.11", - "Programming Language :: Python :: 3.12", - "Programming Language :: Python :: 3.13", - "Operating System :: OS Independent", - ], - ext_modules=external_modules, - setup_requires=requirements, - install_requires=requirements, ) From b78b1b5b882e0ebc009e88b3159dca8cf6c547b6 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 24 Oct 2025 22:21:46 +0000 Subject: [PATCH 3/5] Add automation for dependency management - Add dependabot configuration for automated updates - Add GitHub Actions workflow to validate requirements files - Add pre-commit hook to regenerate requirements on pyproject.toml changes - Update dependency management documentation Co-authored-by: ladc <1212574+ladc@users.noreply.github.com> --- .github/dependabot.yml | 42 +++++++++++++++ .github/workflows/validate_requirements.yml | 54 +++++++++++++++++++ .pre-commit-config.yaml | 10 ++++ .../developer_guide/dependency_management.rst | 24 +++++++-- 4 files changed, 125 insertions(+), 5 deletions(-) create mode 100644 .github/dependabot.yml create mode 100644 .github/workflows/validate_requirements.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 000000000..51c259b4c --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,42 @@ +# Dependabot configuration for automated dependency updates +# https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file + +version: 2 +updates: + # Monitor pip dependencies in pyproject.toml + - package-ecosystem: "pip" + directory: "/" + schedule: + interval: "monthly" + open-pull-requests-limit: 5 + reviewers: + - "pySTEPS/maintainers" + labels: + - "dependencies" + - "python" + commit-message: + prefix: "deps" + include: "scope" + # Group minor and patch updates together + groups: + production-dependencies: + patterns: + - "*" + update-types: + - "minor" + - "patch" + + # Monitor GitHub Actions + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "monthly" + open-pull-requests-limit: 3 + reviewers: + - "pySTEPS/maintainers" + labels: + - "dependencies" + - "github-actions" + commit-message: + prefix: "ci" + include: "scope" diff --git a/.github/workflows/validate_requirements.yml b/.github/workflows/validate_requirements.yml new file mode 100644 index 000000000..373b2c953 --- /dev/null +++ b/.github/workflows/validate_requirements.yml @@ -0,0 +1,54 @@ +name: Validate Requirements Files + +# This workflow ensures that requirements files are up-to-date with pyproject.toml +# It runs on every push and pull request to catch any manual edits to generated files + +on: + pull_request: + paths: + - 'pyproject.toml' + - 'requirements.txt' + - 'requirements_dev.txt' + - 'environment.yml' + - 'environment_dev.yml' + - 'scripts/generate_requirements.py' + push: + branches: + - master + paths: + - 'pyproject.toml' + - 'requirements.txt' + - 'requirements_dev.txt' + - 'environment.yml' + - 'environment_dev.yml' + - 'scripts/generate_requirements.py' + +jobs: + validate-requirements: + runs-on: ubuntu-latest + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Regenerate requirements files + run: | + python scripts/generate_requirements.py + + - name: Check if requirements files are up-to-date + run: | + # Check if there are any differences + if ! git diff --exit-code requirements.txt requirements_dev.txt environment.yml environment_dev.yml; then + echo "❌ Requirements files are out of date!" + echo "" + echo "The generated files don't match pyproject.toml." + echo "Please run 'python scripts/generate_requirements.py' and commit the changes." + exit 1 + else + echo "✅ All requirements files are up-to-date!" + fi diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index cb9b63479..e74715f01 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -4,3 +4,13 @@ repos: hooks: - id: black language_version: python3 + +- repo: local + hooks: + - id: validate-requirements + name: Validate requirements files are up-to-date + entry: python scripts/generate_requirements.py + language: system + files: ^pyproject\.toml$ + pass_filenames: false + always_run: false diff --git a/doc/source/developer_guide/dependency_management.rst b/doc/source/developer_guide/dependency_management.rst index 8061b2edd..4085f3cfb 100644 --- a/doc/source/developer_guide/dependency_management.rst +++ b/doc/source/developer_guide/dependency_management.rst @@ -142,13 +142,27 @@ Version constraints follow these principles: Automated Dependency Updates ----------------------------- -Consider using dependabot or similar tools to: +Pysteps uses GitHub's Dependabot to automatically monitor dependencies for: -- Monitor for security vulnerabilities -- Suggest updates to dependency versions -- Test compatibility with newer versions +- Security vulnerabilities +- Available updates +- Compatibility with newer versions -To enable dependabot, add a ``.github/dependabot.yml`` configuration file. +Dependabot is configured in ``.github/dependabot.yml`` to: + +- Check for updates monthly +- Group minor and patch updates together +- Automatically create pull requests for updates +- Monitor both Python dependencies and GitHub Actions + +When Dependabot creates a pull request: + +1. Review the changes and check the changelog of updated packages +2. Ensure CI tests pass +3. Merge if everything looks good +4. The generated requirements files will be automatically updated via CI + +To manually enable or configure dependabot, edit ``.github/dependabot.yml``. Troubleshooting --------------- From 5dc939974d125d115cd2071ac5ee4ec8a74a2655 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 24 Oct 2025 22:23:49 +0000 Subject: [PATCH 4/5] Fix code review issues - Fix indentation in .pre-commit-config.yaml - Remove team reviewers from dependabot config (may not exist) Co-authored-by: ladc <1212574+ladc@users.noreply.github.com> --- .github/dependabot.yml | 4 ---- .pre-commit-config.yaml | 14 +++++++------- 2 files changed, 7 insertions(+), 11 deletions(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 51c259b4c..3a4a73a16 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -9,8 +9,6 @@ updates: schedule: interval: "monthly" open-pull-requests-limit: 5 - reviewers: - - "pySTEPS/maintainers" labels: - "dependencies" - "python" @@ -32,8 +30,6 @@ updates: schedule: interval: "monthly" open-pull-requests-limit: 3 - reviewers: - - "pySTEPS/maintainers" labels: - "dependencies" - "github-actions" diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index e74715f01..bad1aae83 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -7,10 +7,10 @@ repos: - repo: local hooks: - - id: validate-requirements - name: Validate requirements files are up-to-date - entry: python scripts/generate_requirements.py - language: system - files: ^pyproject\.toml$ - pass_filenames: false - always_run: false + - id: validate-requirements + name: Validate requirements files are up-to-date + entry: python scripts/generate_requirements.py + language: system + files: ^pyproject\.toml$ + pass_filenames: false + always_run: false From bd40a86b16ae78054467db657dcb76b469efaab1 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 24 Oct 2025 22:25:26 +0000 Subject: [PATCH 5/5] Add explicit permissions to validate_requirements workflow - Add contents: read permission to limit GITHUB_TOKEN scope - Addresses security best practice for GitHub Actions Co-authored-by: ladc <1212574+ladc@users.noreply.github.com> --- .github/workflows/validate_requirements.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/validate_requirements.yml b/.github/workflows/validate_requirements.yml index 373b2c953..359e5a9ed 100644 --- a/.github/workflows/validate_requirements.yml +++ b/.github/workflows/validate_requirements.yml @@ -26,6 +26,8 @@ on: jobs: validate-requirements: runs-on: ubuntu-latest + permissions: + contents: read steps: - name: Checkout code