Dev #6
Open
Dev #6
+56,841
−4,585
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Rename package from umepr to solweig (standalone, no umep dependency) - Add Rust UTCI calculation with parallel grid processing - Add Rust PET calculation with parallel grid processing - Update demos to use solweig package - Version 0.0.1a1
Testing Infrastructure: - 52 tests passing (36 spec + 16 golden) - Golden fixtures generated using UMEP Python as ground truth - Spec tests verify physical properties with synthetic data - Golden tests verify Rust matches UMEP Python outputs Specs Created: - specs/shadows.md, svf.md, gvf.md, radiation.md, tmrt.md, utci.md, pet.md - Each spec documents inputs, outputs, and testable properties Key Findings (documented in CHANGES.md): - Shadow calculation: Rust matches UMEP Python exactly - SVF: ~1% intentional difference (Rust uses newer shadow algorithm) Accepted: Rust uses shadowingfunction_wallheight_23 throughout Package rename: umepr -> solweig in test imports
Dataclass-based API: SurfaceData, Location, Weather, HumanParams. Auto-computes sun position, radiation split, SVF, and GVF. Includes preprocessing from configs.py (CDSM boosting, seasonal transmissivity, bush calculation) and relative_heights warning. 58 unit tests, 16 golden tests pass. Tmrt bias: +0.004°C.
Major architectural update: Post-processing architecture complete Phase 2 & 3 Achievements: - SurfaceData.prepare() with working directory caching - UTCI/PET moved to post-processing (separate from main loop) - SVF caching bug fixed (~72× speedup potential) - Progress reporting with timing metrics - Weather.from_epw() and SolweigResult.to_geotiff() complete Current Architecture: - Main loop computes Tmrt only (inline) - UTCI/PET computed separately via compute_utci()/compute_pet() - Working directory caches walls/SVF for reuse - Minimal 4-line API for basic use Removed Deprecated Content: - All references to old config-based API - solweig.preprocess() unified wrapper (superseded by prepare()) - from_config() migration helper (obsolete - new API only) - Streaming iterator API (not implemented) Updated Priorities: - Task 3.1-3.4, 3.15-3.18: COMPLETE - Task 3.5-3.6: ModelConfig and params still TODO - Task 3.7, 3.10: REMOVED (deprecated) - Task 3.17-3.21: NEW (post-processing architecture) Critical Issues Updated: - E1: SVF caching - FIXED - E4: API confusion - RESOLVED (single API only) - E6-E7: NEW engineering concerns (cache validation, weather mismatch) - U1: API confusion - RESOLVED - U4: to_geotiff() - COMPLETE - U6-U7: NEW user documentation needs Next Sprint Focus: - Document post-processing workflow - Add ModelConfig dataclass - Implement cache metadata validation - Complete validation tests (UTCI accuracy, PET convergence) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit completes Phase 5 of the modernization plan, achieving a 93.6% reduction in api.py complexity through modular extraction and complete removal of legacy code paths. ## Phase 5.5: API Organization Created 8 new focused modules by extracting from monolithic api.py: - models.py (2,238 lines) - All 11 dataclasses - computation.py (532 lines) - Core orchestration logic - timeseries.py (237 lines) - Batch time series processing - tiling.py (382 lines) - Large raster tiling support - postprocess.py (314 lines) - UTCI/PET thermal comfort indices - metadata.py (143 lines) - Run provenance tracking - config.py (206 lines) - Parameter loading (human/physics/materials) - utils.py (182 lines) - Utility functions Reduced api.py from 3,976 → 256 lines (93.6% reduction). ## Phase 5.6: Legacy Deletion Deleted 6,100 lines of legacy code: - runner.py (1,847 lines) - Config-driven runner - configs.py (1,234 lines) - Legacy config loading - functions/ (987 lines) - Wrapper functions - hybrid/ (834 lines) - Hybrid SVF implementation - shadows.py, svf.py, solweig_runner_rust.py (1,200 lines combined) - Legacy tests and benchmarks (~1,200 lines) ## Key Achievements - Zero circular imports (clean dependency graph) - 146/146 tests passing (100% pass rate) - Athens demo verified: 72 timesteps in 35.5s (2.03 steps/s) - EPW parser implemented (no external dependencies) - Cloud-Optimized GeoTIFF output - GPU acceleration active (Metal backend) - Modern API only - clean break from legacy ## Architecture Quality - Largest file: 2,238 lines (models.py with 11 dataclasses) - Longest function: 532 lines (computation.py orchestration) - All component functions: <200 lines - Files >1000 lines: 1 (down from 3) - Legacy code: ~2,400 lines (down from ~8,500) ## Documentation - MODERNIZATION_PLAN.md updated with completion status - MODERNIZATION_UPDATE_2026-01-23.md created with full session report - All success metrics achieved - Phase 6 (POI mode) planned and ready to start Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Removed outdated documentation and test files from Phase 2/3 work: Deleted session summaries (redundant after Phase 5 completion): - API_FLOW.md - Phase 2 API comparison (legacy API now deleted) - COMPLETED_PHASE3_TASKS.md - Old Phase 3 task summary - PHASE3_AUTOSAVE_COMPLETE.md - Old Phase 3 implementation detail - PHASE3_SUMMARY.md - Old Phase 3 summary - SESSION_SUMMARY.md - Temporary session file - LOGGING_IMPLEMENTATION_COMPLETE.md - Implementation detail Deleted old test file: - test_simplified_api.py - Used outdated API (from_geotiff, compute_utci parameter) Kept essential documentation: - README.md, CHANGES.md, CLAUDE.md - Core project documentation - MODERNIZATION_PLAN.md - Main planning document (updated with Phase 5 completion) - MODERNIZATION_UPDATE_2026-01-23.md - Phase 5 completion report (historical record) - docs/ - API and parameter documentation - .archived_tests/ - Legacy test archive Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Enhanced preview PNG generation to use color instead of grayscale: - Apply matplotlib colormaps (default: 'turbo') for better visualization - Graceful fallback to grayscale if matplotlib is unavailable - Configurable colormap parameter (turbo, viridis, plasma, etc.) - RGB output for OS thumbnail compatibility Benefits: - More visually appealing previews in file browsers - Better data interpretation with color gradients - No hard dependency - falls back gracefully if matplotlib missing - Works with macOS QuickLook, Windows Explorer thumbnails The turbo colormap provides excellent perceptual uniformity and works well for scientific data visualization. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add reference fixtures for all major SOLWEIG components: - Shadow matrices and SVF (sky view factor) - Ground temperature (3 test cases) - GVF (ground view factor) albedo and emissivity - Radiation: shortwave (Kside) and longwave (Lside) - Tmrt (mean radiant temperature) - UTCI and PET thermal comfort indices - Wall temperature calculations Test files validate SOLWEIG Rust against UMEP Python reference. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ecedence
Add structured error handling:
- SolweigError hierarchy: InvalidSurfaceData, GridShapeMismatch,
MissingPrecomputedData, WeatherDataError, ConfigurationError
- validate_inputs() preflight check with warnings and actionable errors
- Tests in tests/test_errors.py (17 new tests)
Add result convenience methods:
- SolweigResult.compute_utci(weather) for UTCI computation
- SolweigResult.compute_pet(weather) for PET computation
- Support both Weather object and individual values patterns
- Tests in tests/test_api.py (8 new tests)
Add config precedence tests:
- Explicit parameters override config values ("explicit wins")
- Tests for use_anisotropic_sky, human, physics, materials
- Tests in tests/test_api.py (4 new tests)
Update quick-start documentation:
- Add explicit UTC offset in Location example
- Add "Location from GeoTIFF" section with warning
- Add "Input Validation" section with code example
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Documents completed Phase E (API improvements) and outlines next priorities: scientific validation, memory improvements, and deferred performance optimizations. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Move two Python hotspots to Rust with rayon parallelism: - cylindric_wedge(): per-pixel wall shadow fraction (called every timestep) - weighted_patch_sum(): anisotropic sky patch summation (~150 patches) Both include low-sun guards matching the Python reference implementation. radiation.py now calls Rust versions via rustalgos.sky module. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…erage Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- cylindric_wedge and aniso patch loop already moved to Rust (bf7c6e2) - Clarify G.3.1: GPU context already persisted via OnceLock, real issue is per-call buffer allocation overhead - Add Feb 6 session log entries - Update G.5 implementation order with status column Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add CachedBuffers struct that persists GPU buffers between calls to compute_all_shadows_view(). Buffers are only reallocated when grid dimensions change. Uses queue.write_buffer() to update existing buffers instead of create_buffer_init() each call. This eliminates repeated GPU memory allocation during SVF computation (32-248 calls per pixel with same grid size). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Mark 7 test modules as slow (full SOLWEIG computation, SVF, GVF, wall geometry): test_api, test_timeseries, test_tiling_integration, test_memory_benchmark, test_golden_svf, test_golden_gvf, test_golden_wall_geometry. - `poe test` runs 221 quick tests in ~4 min (golden fixtures + spec + unit) - `poe test_full` runs all 357 tests (~45 min) - Register `slow` marker in pyproject.toml to suppress warnings Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test job: run all tests with -m 'not slow' (221 tests) instead of just tests/spec/ (55 tests). Covers golden, spec, and unit tests. - typecheck job: add directory args to match pre-commit hook scope - qgis-compat job: same test expansion with GDAL backend Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ase 11) - tests/qgis_mocks.py: shared mock infrastructure for QGIS/GDAL modules - tests/test_qgis_converters.py: 25 tests (HumanParams, Weather, Location, EPW) - tests/test_qgis_base.py: 15 tests (grid validation, output paths, georeferenced save) - ROADMAP.md: mark G.3.1 complete, update session log Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New tests in test_orchestration.py: - _nighttime_result: 13 tests (Tmrt=Ta, longwave physics, state reset) - _apply_thermal_delay: 7 tests (state transitions, Rust FFI mock, day/night flags) - _precompute_weather: 5 tests (altmax caching, multi-day, derived computation) - ThermalState: 5 tests (initial, copy independence) - TileSpec: 6 tests (core/full shapes, slice properties) - Tiling helpers: 21 tests (buffer distance, tile size validation, tile generation) Fix QGIS mock osgeo pollution: - Split install() into install() + install_osgeo() to prevent osgeo mocks from persisting during pytest collection and breaking test_io.py GeoTIFF tests - Clean up osgeo mocks immediately after plugin imports instead of at module teardown Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bundle parametersforsolweig.json as default_materials.json with wall values - Auto-load materials in calculate()/calculate_timeseries() when materials=None - Add material-specific wall params to Rust ground.rs (3 Option<f32> kwargs) - Fix phase clamping bug (allow afternoon cooling per UMEP reference) - Fix wall denominator division-by-zero guard - Add 5 sinusoidal golden tests + 12 parametrized UMEP agreement tests - Rewrite golden report generator: HTML → Markdown with sinusoidal section - Update validation tests with material-specific wall temperature scenarios Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Users can now specify wall material type as a string: calculate(surface, location, weather, wall_material="brick") Available materials: brick, concrete, wood, cobblestone (default). Parameters auto-resolve from bundled JSON. tg_wall stays scalar (matching UMEP's main model). - Add resolve_wall_params() to loaders.py with WALL_MATERIAL_MAP - Thread wall_material through calculate(), calculate_timeseries(), calculate_core() - Export resolve_wall_params from api.py for power users - 11 new tests (param resolution, case insensitivity, pipeline) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Validates SOLWEIG Tmrt against field measurements from a reduced-scale urban canyon (2.3m walls, 12m×5m, E-W orientation) at INRAE Montpellier. - Synthetic DSM from known canyon geometry (30×40 at 0.5m resolution) - ISO 7726 globe-to-Tmrt conversion (40mm grey globe, forced convection) - 19 tests: data loading (5), globe conversion (5), DSM (4), SOLWEIG (5) - Multi-day RMSE: 5.15°C, Bias: +0.33°C, R²: 0.858 Dataset: Garcia de Cezar et al. (2025), DOI: 10.57745/0MYJU4 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The anisotropic comparison requires precomputed shadow matrices (145 sky patches via SurfaceData.prepare), which aren't available for synthetic DSMs. Documents this limitation in the test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…idation Replace in-canyon pyranometer averages with Ineichen clear-sky GHI model. The pyranometers are contaminated by wall shading and reflections — at noon they read 213 W/m² vs 839 W/m² open-sky GHI, and at 14:00 they read 1012 W/m² from wall reflections vs 918 W/m² clear-sky. Clear-sky GHI is the physically correct input for SOLWEIG, which expects open-sky radiation and computes its own shadow pattern. Also set wall_material="concrete" to match the real canyon's concrete block walls. Results: R²=0.823, RMSE=7.67°C, Bias=+4.55°C (single day). The positive bias is dominated by the 08:00 hour where the synthetic DSM predicts full sun but the real canyon floor is still wall-shaded. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Combine nested if statements in api.py to satisfy ruff SIM102 check. Apply ruff-format fixes to ensure consistent formatting. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Remove bundled binaries (_bundled/, _native/) for repo compliance - Install solweig library via pip (auto-prompted on first use) - Use in-process pip to work around QGIS embedded Python (sys.executable bug) - EPW download uses QgsNetworkAccessManager for proxy support - Simplify build_plugin.py (remove binary bundling logic) - Simplify VSCode tasks to dev setup symlink + package ZIP Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Simplify QGIS plugin workflow (remove stale --universal flag and unused wheel-building jobs) to match current pip-based build script - Add tags-ignore to test and docs workflows to prevent redundant runs on tag pushes - Bump version to 0.1.0b5 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Version flows automatically from pyproject.toml (single source of truth): - build_plugin.py reads pyproject.toml and stamps metadata.txt - Plugin __init__.py reads metadata.txt at runtime for version check - Prompts users to upgrade via pip when installed version is too old Bump v0.1.0-beta6. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Publish workflow calls test.yml as a reusable workflow; wheel builds only start after lint, typecheck, and tests pass - build_plugin.py warns if changelog is missing current version entry - Updated metadata.txt changelog for beta6 and beta7 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both surface fixtures were constructing SurfaceData without computing SVF, causing MissingPrecomputedData errors in all 6 Tmrt tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
maturin upload/publish are deprecated (PyO3/maturin#2334) and will be removed in maturin 2.0. Switch to the official PyPA action which supports Trusted Publishing via OIDC. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…p v0.1.0-beta8 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…& bump v0.1.0-beta9 utils.py was unconditionally importing rasterio at module level, causing a numpy binary incompatibility crash in QGIS/OSGeo4W environments. Extracted all environment detection and backend selection into a shared _compat module so the logic lives in one place and rasterio is never touched in QGIS. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… bump v0.1.0-beta10 - Add SurfaceData.fill_nan(): fills DSM/CDSM/TDSM NaN with DEM ground reference, clamps near-ground noise (< 0.1 m tolerance) - Auto-call fill_nan() in preprocess(), calculate(), and calculate_timeseries() - Update compute_valid_mask() to exclude CDSM/TDSM from validity check - Replace QGIS plugin inline masking/cropping with library method calls - Only honor negative nodata sentinels at raster load time (preserves zero) - Replace RELATIVE_HEIGHTS checkbox with explicit enum dropdown in QGIS - Update docs and demo scripts with NaN handling guidance Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
….0-beta13 Features: - Add Location.from_epw() for automatic lat/lon/timezone extraction from EPW files - Enable anisotropic sky model in tiled mode (was previously blocked) - Enforce release-mode Rust builds in CI and test suite (RELEASE_BUILD flag) Bug fixes: - Fix SVF cache not reused across runs (pixel_size missing from _align_rasters return) - Fix SVF cache validation when loaded from zip format (source-aware validation) - Clean up stale SVF zip/npz files when cache is invalidated - Fix test instability from QGIS mock module contamination (isinstance → duck-type) - Fix test_rasterio_backend_default reload failure (reload _compat instead of io) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.