Add support for downloading ERA5 pressure levels data product#93
Closed
ewquon wants to merge 55 commits intoNumericalEarth:mainfrom
Closed
Add support for downloading ERA5 pressure levels data product#93ewquon wants to merge 55 commits intoNumericalEarth:mainfrom
ewquon wants to merge 55 commits intoNumericalEarth:mainfrom
Conversation
…ressureLevels
- Updated exports to include ERA5HourlyPressureLevels, ERA5MonthlyPressureLevels, ERA5_all_pressure_levels, pressure_field
- Added using Statistics, Oceananigans.Fields.CenterField/interior, Oceananigans.BoundaryConditions.fill_halo_regions!, native_grid, InverseGravity to imports
- Extended import block to include is_three_dimensional, reversed_vertical_axis, conversion_units
- Added ERA5_all_pressure_levels constant (37 standard hPa levels)
- Added ERA5PressureDataset, ERA5HourlyPressureLevels, ERA5MonthlyPressureLevels with keyword constructors
- Added ERA5PressureMetadata{D} and ERA5PressureMetadatum type aliases
- Added Base.size, all_dates, is_three_dimensional, reversed_vertical_axis dispatches
- Added ERA5PL_dataset_variable_names and ERA5PL_netcdf_variable_names dicts (15 variables each)
- Added available_variables, dataset_variable_name, netcdf_variable_name, conversion_units dispatches
- Added retrieve_data(::ERA5PressureMetadatum) — reads 4D NetCDF, reverses vertical axis
- Added metadata_prefix(::ERA5PressureMetadata) — uses ERA5PL_dataset_variable_names for filename construction
- Added _std_atm_geopotential_height, _std_atm_z_interfaces, z_interfaces(::ERA5PressureMetadata)
- Added pressure_field and mean_geopotential_heights
glwagner
reviewed
Mar 4, 2026
glwagner
reviewed
Mar 4, 2026
glwagner
reviewed
Mar 4, 2026
glwagner
reviewed
Mar 4, 2026
glwagner
reviewed
Mar 4, 2026
glwagner
reviewed
Mar 4, 2026
glwagner
reviewed
Mar 4, 2026
glwagner
reviewed
Mar 4, 2026
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
glwagner
reviewed
Mar 4, 2026
glwagner
reviewed
Mar 4, 2026
glwagner
reviewed
Mar 4, 2026
…s, without explicitly defining metadata
Combined CDSAPI requests will return a single combined netcdf; set cleanup=false to keep the "_tmp_multi_<datetime>_*" files
Note: inpainting is now turned OFF for 2D data -- the reanalysis data should be complete and turning on inpainting would artificially fill in data that should be masked (e.g., ocean quantities over land) Tested for 2D (single levels) and 3D (pressure levels)
to disambiguate from ERA5PressureLevels*
…to eq/era5_pressure_levels
including instantaneous, accumulated, and averaged
…th.jl into eq/era5_pressure_levels
…to eq/era5_pressure_levels Adopt the bbox→region refactor (NumericalEarth#142): rename `metadata.bounding_box` to `metadata.region` across ERA5 + CDSAPI extension code, examples, and tests. Drop the obsolete `location(::ERA5Metadata)` override in favor of the new generic `dataset_location` dispatch, and the redundant per-Metadatum `metadata_filename` override (the generic version returns the stored filename). Adopt `reversed_latitude_axis(::ERA5Dataset) = true` for the new column-field path's φ-coord flip; ERA5's `retrieve_data` overrides keep their manual `reverse(data, dims=2)` since they bypass the generic path. Restore the CDS API environment-setup docstring on `download_dataset(::ERA5Metadatum)`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Upstream's column-field test used `ERA5Hourly`, which this branch renamed to `ERA5HourlySingleLevel` (reserving the simpler name for the abstract dataset family). Update the test accordingly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…to eq/era5_pressure_levels
Per @glwagner's review: a `Field{Nothing, Nothing, Center}` represents the same per-level pressure values as the previous `CenterField` but without copying across the full horizontal grid. Use `set!` for the column assignment, drop the per-k loop, and let the field's eltype follow the grid (no more hardcoded `Float32`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two stopgaps to keep CI green when CDSAPI credentials aren't configured: - Surface-level tests: extend `download_dataset_with_fallback` with a fallback-of-the-fallback. If both CDS and the NumericalEarthArtifacts download fail (e.g., the artifact key was renamed when ERA5 dataset types were renamed), it now returns `false` instead of throwing. Callers `|| return` to skip the testset cleanly with an `@info` rather than cascading file-not-found errors. - Pressure-level tests: two subtests (`Download and 3D Field`, `Geopotential height conversion`) still call `download_dataset` directly without the fallback wrapper, so guard them with a credentials check that early-returns with an `@info`. TODO comment references the long-term fix (upload pressure-level test files to NumericalEarthArtifacts and switch to the fallback wrapper). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surface-level testsets each downloaded `2m_temperature` then removed it, forcing the next testset to re-download the same bytes — three CDS round-trips for one fixture. Keep the pre-clean only in the testset that's testing the download path itself; let downstream testsets reuse the file and run cleanup in the last consumer. Pressure-level "Geopotential height conversion" pre-cleaned the `geopotential_...nc` that the previous testset's `z_interfaces` side-effect leaves on disk, then re-downloaded it. Drop the pre-clean and the redundant explicit `download_dataset(meta_z)` call (Field() already downloads if needed). Net: surface round-trips drop 3 → 1, geopotential round-trips drop 2 → 1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous commit (c6653b2) folded the skip behavior directly into download_dataset_with_fallback by changing its return type from the result of download_fn() to a Bool. That broke other callers — JRA55 tests do `fts = download_dataset_with_fallback(...) do ... end; @test isfile(fts.path)`, which silently NPEs when fts becomes Bool. Restore download_dataset_with_fallback to its original semantics (returns download_fn() result, throws on double-failure). Add a new try_download_or_skip wrapper that catches the double-failure case, emits @info + CI warning, and returns false — used only by the CDS surface tests via `try_download_or_skip(...) do ... end || return`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
@simone-silvestri I'd like to update NumericalEarthArtifacts:
Let me know how to proceed. |
Member
|
Note: existing artifacts should never be removed or replaced, otherwise existing tests will fail in the future. For example the renaming operation should be a copy. |
Collaborator
Author
|
CI passes in #197, closing here. |
Member
|
Sounds good, maybe we can still add the artifacts to NumericalEarthArtifacts for future users that fork the repo (the instructions should be in the README) |
Collaborator
Author
|
Will do! Didn't know if I had write access or not. |
Collaborator
Author
@simone-silvestri Looks like I don't have push permissions % gh release upload data-v1 2m_temperature_ERA5HourlySingleLevel_2005-02-16T12_0.0_5.0_40.0_45.0.nc --repo NumericalEarth/NumericalEarthArtifacts
HTTP 404: Not Found (https://uploads.github.com/repos/NumericalEarth/NumericalEarthArtifacts/releases/284970955/assets?label=&name=2m_temperature_ERA5HourlySingleLevel_2005-02-16T12_0.0_5.0_40.0_45.0.nc) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To address #88