-
Notifications
You must be signed in to change notification settings - Fork 7
Add support for downloading ERA5 pressure levels data product (new) #197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
62 commits
Select commit
Hold shift + click to select a range
4078d8c
Implement ERA5PressureDataset, ERA5HourlyPressureLevels, ERA5MonthlyP…
ewquon 54a2a0c
Add ERA5*PressureLevels exports
ewquon 8fede16
Add InverseGravity for geopotential height calc
ewquon 13e4fab
Add ERA5 pressure-levels downloader
ewquon afdd227
Add CDS tests, verified 95/95 pass
ewquon 2cb7141
Merge branch 'main' into eq/era5_pressure_levels
ewquon c09f6da
Fix imports
ewquon 9c48937
Add day, hour to date_str for filenaming
ewquon aec39c8
Allow simultaneous download of multiple variables
ewquon 03efebc
Aggregate hourly downloads into API requests per calendar day
ewquon f1de9e5
Cleanup file naming
ewquon 916614e
Cleanup, clarify
ewquon 8f42dee
Allow one or more variables to be downloaded, at one or more datetime…
ewquon 3cac1f8
Add cleanup flag for multi-datetime downloads
ewquon c56c6fc
Fix silly Claude vertical-level flip; no inpainting needed for ERA5
ewquon e552ccb
Fix handling of ERA5 latitude grid
ewquon fce1404
Rename original "ERA5*" metadata to "ERA5SingleLevels*"
ewquon 4643ffa
Fix ERA5 grid creation
ewquon 9089c99
Fix bbox handling
ewquon cf2b8e0
Use standard units for pressure_levels
ewquon 4139a6e
No special treatment of lower z_interface, allow z < 0
ewquon c45200a
Fix defaults
ewquon 18cba4c
Implement mean_geopotential_z_interfaces; make mutable to allow z update
ewquon ae0f0b1
Organize ERA5 code into single-level and pressure-levels source files
ewquon c5c2e2f
Use consistent ERA5 naming
ewquon 77d33f3
Automatic geopotential download to calculate mean geopotential heights
ewquon 5aa675b
Streamline automatic download
ewquon 79dcb17
Refactor; add examples
ewquon 3b44aba
Merge branch 'main' of github.com:NumericalEarth/NumericalEarth.jl in…
ewquon 577b3c4
Address a very very very minor point
ewquon f4afe09
Retrieve var name from correct dict
ewquon c28db4f
Add additional surface variables
ewquon 74cb622
Improved readability
ewquon ad135e3
Follow Oceananigans convention
ewquon b6baf64
Aesthetics
ewquon a0206b2
Merge branch 'main' of github.com:NumericalEarth/NumericalEarth.jl in…
ewquon 7cc877e
Handle download of instantaneous and accumulated/mean quantities
ewquon 567101b
Remove redundant duplicate date when requesting ERA5 single-level hou…
ewquon 48a6b3e
Add mean evaporation rate aka "time-mean moisture flux" [kg m⁻² s⁻¹]
ewquon 9878169
Merge branch 'main' of github.com:NumericalEarth/NumericalEarth.jl in…
ewquon 0eb2c78
Most robustly handle situations with multiple data types in one request
ewquon 7b7d1c7
Better handle geopotential height variability
ewquon ae169b9
Allow use of pre-downloaded ERA5 data without depending on CDSAPI
ewquon 4afc729
Fix bug introduced by prev commit; add additional era5 vars
ewquon d21e0be
Merge branch 'main' of https://github.com/NumericalEarth/NumericalEar…
ewquon 0960d35
Minor cleanup
ewquon d6dc0ca
Merge branch 'main' of github.com:NumericalEarth/NumericalEarth.jl in…
ewquon 50cc57d
Fix ERA5 Column grid test to use renamed type
ewquon 294db53
Merge branch 'main' of github.com:NumericalEarth/NumericalEarth.jl in…
ewquon bb099f0
Simplify pressure_field to a reduced column Field
ewquon 575b52d
Temporarily define hPa
ewquon a597c60
Cleanup CDS download test
ewquon be9657c
Share CDS test fixtures across testsets to avoid redundant downloads
ewquon 6e6e84c
Support Column regions for ERA5 CDS downloads
ewquon c7ad02d
Add network-free tests for CDSAPIExt dispatch and NetCDF helpers
ewquon f70a99c
Merge branch 'main' of github.com:NumericalEarth/NumericalEarth.jl in…
ewquon 5eacbc7
Cleanup
ewquon eeb699a
Add unit-level tests for ERA5 single-level helpers
ewquon 445f493
Add tests for calendar-day grouping and skip_existing short-circuit
ewquon e8a77a3
Merge branch 'main' of github.com:NumericalEarth/NumericalEarth.jl in…
ewquon 30b9560
Add unit tests for pure helpers in metadata_field, ERA5_pressure_leve…
ewquon bdc98d7
Apply suggestion from @glwagner
ewquon File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| # Download 3-D snapshots of the atmospheric state from ERA5, e.g., to be used with FieldTimeSeries | ||
| # | ||
| # ## Install dependencies | ||
| # | ||
| # ```julia | ||
| # using Pkg | ||
| # pkg"add NumericalEarth CDSAPI" | ||
| # ``` | ||
| # | ||
| # You also need CDS API credentials in `~/.cdsapirc`. | ||
| # See <https://cds.climate.copernicus.eu/how-to-api> for setup instructions. | ||
| # | ||
| # See <https://cds.climate.copernicus.eu/datasets/reanalysis-era5-pressure-levels?tab=download> for | ||
| # more information about pressure-level data | ||
|
|
||
| using NumericalEarth | ||
| using NumericalEarth.DataWrangling: Metadata, BoundingBox, download_dataset | ||
| using NumericalEarth.DataWrangling.ERA5 | ||
| using CDSAPI | ||
| using Dates | ||
| using Oceananigans | ||
|
|
||
| using Statistics | ||
| using CairoMakie | ||
|
|
||
| selected_levels = filter(≥(250hPa), ERA5_all_pressure_levels) # select all levels below 250 hPa | ||
| dataset = ERA5HourlyPressureLevels(selected_levels) | ||
|
|
||
| dates = DateTime(2004, 12, 16):Hour(1):DateTime(2005, 01, 09) # van Zanten et al 2011 study period | ||
|
|
||
| # Rauber et al 2007, Fig 1: focus region | ||
| region = BoundingBox(latitude=(17, 18.5), longitude=(-62.5, -61)) | ||
|
|
||
| # OPTIONAL: download all variables at once (fewer CDS API requests) | ||
| variables = [:geopotential, # for calculating zlevels as the mean geopotential height | ||
| :eastward_velocity, | ||
| :northward_velocity, | ||
| :temperature, | ||
| :specific_humidity] | ||
| download_dataset(variables, dataset, dates; region) | ||
|
|
||
| # ## Create time series | ||
|
|
||
| T_meta = Metadata(:temperature; dataset, dates, region) | ||
| q_meta = Metadata(:specific_humidity; dataset, dates, region) | ||
| u_meta = Metadata(:eastward_velocity; dataset, dates, region) | ||
| v_meta = Metadata(:northward_velocity; dataset, dates, region) | ||
|
|
||
| T_series = FieldTimeSeries(T_meta; time_indices_in_memory=2) | ||
| q_series = FieldTimeSeries(q_meta; time_indices_in_memory=2) | ||
| u_series = FieldTimeSeries(u_meta; time_indices_in_memory=2) | ||
| v_series = FieldTimeSeries(v_meta; time_indices_in_memory=2) | ||
|
|
||
| # ## Vertical profiles (mean ± spread) | ||
|
|
||
| # Height coordinate (m) from the grid | ||
| z = znodes(T_series[1]) | ||
| Nz = length(z) | ||
| Nt = length(T_series.times) | ||
|
|
||
| # Pressure at each level (hPa), ordered bottom-to-top (k=1 ⇒ highest pressure) | ||
| p_levels = sort(selected_levels, rev=true) ./ hPa # Pa → hPa | ||
|
|
||
| # Horizontal-mean profile for each snapshot → (Nz, Nt) arrays | ||
| function horizontal_mean_profiles(series) | ||
| profiles = zeros(Nz, Nt) | ||
| for n in 1:Nt | ||
| data = interior(series[n], :, :, :) | ||
| profiles[:, n] = mean(data, dims=(1, 2)) | ||
| end | ||
| return profiles | ||
| end | ||
|
|
||
| T_profiles = horizontal_mean_profiles(T_series) | ||
| q_profiles = horizontal_mean_profiles(q_series) | ||
| u_profiles = horizontal_mean_profiles(u_series) | ||
| v_profiles = horizontal_mean_profiles(v_series) | ||
|
|
||
| # Convert T → potential temperature: θ = T (p₀/p)^(R/cₚ) | ||
| Rₐ_over_cₚ = 0.286 | ||
| θ_profiles = T_profiles .* (1000 ./ p_levels) .^ Rₐ_over_cₚ | ||
|
|
||
| # Convert specific humidity from kg/kg → g/kg | ||
| q_profiles .*= 1000 | ||
|
|
||
| # ## Plot | ||
|
|
||
| fig = Figure(size=(900, 500), fontsize=12) | ||
|
|
||
| ax_θ = Axis(fig[1, 1], xlabel="θ [K]", ylabel="Height [m]") | ||
| ax_q = Axis(fig[1, 2], xlabel="qᵥ [g kg⁻¹]", ylabel="Height [m]") | ||
| ax_u = Axis(fig[1, 3], xlabel="u [m s⁻¹]", ylabel="Height [m]") | ||
| ax_v = Axis(fig[1, 4], xlabel="v [m s⁻¹]", ylabel="Height [m]") | ||
|
|
||
| for (ax, profiles) in [(ax_θ, θ_profiles), | ||
| (ax_q, q_profiles), | ||
| (ax_u, u_profiles), | ||
| (ax_v, v_profiles)] | ||
| μ = vec(mean(profiles, dims=2)) | ||
| #lo = vec(minimum(profiles, dims=2)) | ||
| #hi = vec(maximum(profiles, dims=2)) | ||
| lo = [quantile(r, 0.25) for r in eachrow(profiles)] | ||
| hi = [quantile(r, 0.75) for r in eachrow(profiles)] | ||
| band!(ax, z, lo, hi; direction=:y, color=(:gray, 0.4)) | ||
| lines!(ax, μ, z; color=:black, linewidth=2) | ||
| end | ||
|
|
||
| xlims!(ax_θ, 293, 317) | ||
| xlims!(ax_q, 0, 20.4) | ||
| xlims!(ax_u, -10, 0) | ||
| xlims!(ax_v, -10, 0) | ||
|
|
||
| linkyaxes!(ax_θ, ax_q, ax_u, ax_v) | ||
| ylims!(ax_θ, 0, 4000) | ||
|
|
||
| hideydecorations!(ax_q) | ||
| hideydecorations!(ax_u) | ||
| hideydecorations!(ax_v) | ||
|
|
||
| save("ERA5_pressure_level_profiles.png", fig; px_per_unit=4) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| # Download 3-D snapshots of the atmospheric state from ERA5, e.g., to be used with FieldTimeSeries | ||
| # | ||
| # ## Install dependencies | ||
| # | ||
| # ```julia | ||
| # using Pkg | ||
| # pkg"add NumericalEarth CDSAPI" | ||
| # ``` | ||
| # | ||
| # You also need CDS API credentials in `~/.cdsapirc`. | ||
| # See <https://cds.climate.copernicus.eu/how-to-api> for setup instructions. | ||
| # | ||
| # See <https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels?tab=download> for | ||
| # more information about single-level data | ||
|
|
||
| using NumericalEarth.DataWrangling: Metadata, BoundingBox, download_dataset | ||
| using NumericalEarth.DataWrangling.ERA5 | ||
| using CDSAPI | ||
| using Dates | ||
| using Oceananigans | ||
|
|
||
| using Statistics: mean | ||
| using CairoMakie | ||
| #precip_cmap = :rain | ||
| precip_cmap = cgrad([:indigo, :darkblue, :blue, :deepskyblue, :cyan, | ||
| :palegreen, :green, :yellow, :orange, :red, :pink]) | ||
|
|
||
|
|
||
| dataset = ERA5HourlySingleLevel() | ||
|
|
||
| #dates = DateTime(2004, 12, 16):Hour(1):DateTime(2005, 01, 09) # van Zanten et al 2011 study period | ||
| dates = DateTime(2004, 12, 16):Hour(1):DateTime(2004, 12, 23) # shorter time range for demo | ||
|
|
||
| # Rauber et al 2007, Fig 1: precip map | ||
| region = BoundingBox(latitude=(-25, 35), longitude=(-110, 30)) | ||
|
|
||
| # OPTIONAL: download all variables at once (fewer CDS API requests) | ||
| # variables = [:total_precipitation, | ||
| # :sea_surface_temperature, | ||
| # :temperature, # at 2 m | ||
| # :dewpoint_temperature, # at 2 m | ||
| # :surface_pressure] | ||
| # download_dataset(variables, dataset, dates; region) | ||
|
|
||
| # ## Create time series | ||
|
|
||
| precip_meta = Metadata(:total_precipitation; dataset, dates, region) | ||
| precip_series = FieldTimeSeries(precip_meta) | ||
|
|
||
| # ## Plotting | ||
|
|
||
| Nt = length(precip_series.times) | ||
| λ, φ, _ = nodes(precip_series[1]) | ||
|
|
||
| # ERA5 total_precipitation is in metres per hour; convert to mm/day | ||
| to_mm_day = 1000 * 24 | ||
|
|
||
| # ### Time-averaged precipitation map | ||
|
|
||
| precip_avg = mean(interior(precip_series[n], :, :, 1) for n in 1:Nt) .* to_mm_day | ||
|
|
||
| fig1 = Figure(size=(900, 400)) | ||
| ax1 = Axis(fig1[1, 1], | ||
| title = "Mean precipitation (2004-12-16 to 2005-01-08)", | ||
| xlabel = "Longitude (°)", | ||
| ylabel = "Latitude (°)") | ||
| ax1.xticks = -90:30:30 | ||
| hm = heatmap!(ax1, λ, φ, precip_avg; colormap=precip_cmap, colorrange=(0, 12)) | ||
| Colorbar(fig1[1, 2], hm, label="Precipitation (mm/day)") | ||
| save("ERA5_mean_precipitation.png", fig1) | ||
|
|
||
| # ### Precipitation animation | ||
|
|
||
| fig2 = Figure(size=(900, 400)) | ||
|
|
||
| n = Observable(1) | ||
| precip_n = @lift interior(precip_series[$n], :, :, 1) .* to_mm_day | ||
| anim_title = @lift "Precipitation (mm/day), " * string(first(dates) + Second(round(Int, precip_series.times[$n]))) | ||
|
|
||
| ax2 = Axis(fig2[1, 1], | ||
| title = anim_title, | ||
| xlabel = "Longitude (°)", | ||
| ylabel = "Latitude (°)") | ||
| ax2.xticks = -90:30:30 | ||
|
|
||
| hm2 = heatmap!(ax2, λ, φ, precip_n; colormap=precip_cmap, colorrange=(0, 12)) | ||
| Colorbar(fig2[1, 2], hm2, label="Precipitation (mm/day)") | ||
|
|
||
| record(fig2, "ERA5_precipitation.mp4", 1:Nt; framerate=12) do nn | ||
| n[] = nn | ||
| end | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.