Method to enforce "predownloading" of data, when desired

[glwagner]

Problem

Currently, datasets are downloaded on demand when Field(metadata), FieldTimeSeries(metadata), or DatasetRestoring(metadata) is called and data is not already present. This method is useful for writing scripts that will "work anywhere" on the first call without extra steps (eg downloading necessary data from somewhere, which might requiring diving into documentation...)

However, for expensive applications it can be useful to enforce the opposite; that a script will not run unless the data has already been downloading. Similarly, it'd be convenient to be able to take a script written by someone else, and automatically download all of the data it needs (eg on a login node) before submitting a job to a cluster. In other words we want

1. Pre-downloading for cluster jobs: When running simulations on a cluster, data should be downloaded on a login node before submitting a Slurm job, to avoid wasting compute resources (and because compute nodes may lack internet access).

2. Guarding against accidental downloads: For expensive or production simulations, there should be a way to guarantee that all data is already on disk — and error immediately if it isn't — rather than silently downloading mid-simulation.

Proposed solution

Three layers, from low-level to high-level:

1. NUMERICALEARTH_DATA environment variable

Control download behavior at runtime:

Value	Behavior
unset / "auto"	Download on demand (current default)
"existing"	Error if data isn't already on disk — never download

Analogous to Julia's --pkgimages=existing. Usage:

NUMERICALEARTH_DATA=existing srun julia --project run_simulation.jl

Implementation: a check at the top of each download_dataset method that verifies files exist when in existing mode, and errors with a clear message listing the missing files.

2. Metadata extraction from high-level constructors

Refactor constructors like ERA5PrescribedAtmosphere so their metadata-creation logic is accessible separately:

# Returns Vector{Metadata} for all 8 ERA5 variables
metadata = ERA5PrescribedAtmosphere_metadata(;
    dataset = ERA5Hourly(),
    start_date = DateTime(2020, 1, 1),
    end_date = DateTime(2020, 12, 31),
    region = BoundingBox(longitude=(200, 220), latitude=(35, 55)))

This enables constructing metadata (cheap, no I/O) without building fields.

A convenience function for batch downloading:

download_datasets(metadata)               # Vector{Metadata}
download_datasets(metadata1, metadata2...) # varargs

3. DataManifest.toml for declarative data requirements

A TOML manifest declaring all data a simulation needs, living alongside Project.toml:

# DataManifest.toml

[[atmosphere]]
type = "ERA5PrescribedAtmosphere"
dataset = "ERA5Hourly"
start_date = "2020-01-01"
end_date = "2020-12-31"
longitude = [200, 220]
latitude = [35, 55]

[[datasets]]
variable = "temperature"
dataset = "GLORYSDaily"
start_date = "2020-01-01"
end_date = "2020-12-31"
longitude = [200, 220]
latitude = [35, 55]

Two directions:

• Code → Manifest: NumericalEarth.write_manifest("DataManifest.toml", metadata_collection) — export data requirements from simulation setup code
• Manifest → Downloads: NumericalEarth.download_datasets("DataManifest.toml") — download without importing simulation code

This is especially useful when simulation setups are implemented in source code (packages) rather than scripts, and for automated systems (CI/CD, deployment tooling) that need to act on data requirements without running Julia simulation code.

Example cluster workflow

# download_data.jl — run on login node
using NumericalEarth

atm = ERA5PrescribedAtmosphere_metadata(;
    start_date = DateTime(2020, 1, 1),
    end_date = DateTime(2020, 12, 31),
    region = BoundingBox(longitude=(200, 220), latitude=(35, 55)))

ocean_T = Metadata(:temperature; dataset=GLORYSDaily(),
                    start_date = DateTime(2020, 1, 1),
                    end_date = DateTime(2020, 12, 31))

download_datasets(atm..., ocean_T)

# On login node:
julia --project download_data.jl

# Submit job with offline guard:
NUMERICALEARTH_DATA=existing sbatch run_simulation.sh

Implementation order

1. NUMERICALEARTH_DATA=existing guard (small, high value)
2. Metadata extraction functions + download_datasets convenience (enables pre-download scripts)
3. DataManifest.toml read/write (enables automation and infrastructure integration)

[glwagner]

cc @awiteck @jsheedy curious your thoughts about this ENV variable.

I would also like to explore the possibility of somehow analyzing scripts and automatically downloading data. But Claude so far rebuffed my attempts to conceive of such as system. So we are stuck with the need to build "auxiliary download" scripts like download_data.jl above.

[glwagner]

Refined design: DryRun() architecture + NUMERICALEARTH_MODE env var

After further brainstorming, here's a more concrete design.

Problem with separate download scripts

Maintaining a separate download_data.jl script is error-prone and tedious. Simulation parameters (dates, regions, datasets) get duplicated, and the download script drifts out of sync as the simulation evolves. We want one script for both downloading and running.

DryRun() architecture

A new architecture type that makes a simulation script "safe to run" on a login node:

struct DryRun <: AbstractSerialArchitecture end

DryRun() does three things:

1. Behaves as CPU for all Oceananigans dispatch (array allocation, device operations, etc.)
2. Coarsens grids automatically — LatitudeLongitudeGrid(DryRun(); size=(1440, 720, 50), ...) returns a tiny grid (e.g., size=(4, 4, 2)). This is critical because a production-resolution grid may only be constructible in a Distributed setting.
3. Signals to run!/time_step! to skip actual time-stepping.

Implementation is manageable — DryRun <: AbstractSerialArchitecture inherits ~31 methods from abstract types. The CPU-specific methods (about a dozen) are trivial forwards:

device(::DryRun) = KernelAbstractions.CPU()
array_type(::DryRun) = Array
on_architecture(::DryRun, a::Array) = a
# ... etc

Grid constructors are extended in NumericalEarth to coarsen:

function LatitudeLongitudeGrid(::DryRun; size, kw...)
    dry_run_size = map(s -> min(s, 4), size)
    return LatitudeLongitudeGrid(CPU(); size=dry_run_size, kw...)
end

# Same for RectilinearGrid, TripolarGrid, ORCAGrid, ImmersedBoundaryGrid

Field and FieldTimeSeries constructors download but skip data loading:

function Field(metadata::Metadatum, ::DryRun; kw...)
    download_dataset(metadata)       # real download
    grid = native_grid(metadata, CPU(); dry_run=true)  # tiny grid
    LX, LY, LZ = location(metadata)
    return Field{LX, LY, LZ}(grid)  # zero-filled, no data load or inpainting
end

And time_step! is a no-op:

function time_step!(model::EarthSystemModel{<:Any, <:Any, <:Any, <:Any, <:Any, <:DryRun}, Δt; kw...)
    @info "DryRun mode: skipping time-stepping. All datasets downloaded."
    return nothing
end

NUMERICALEARTH_MODE environment variable

A single env var spanning both use cases:

Value	Behavior
unset (default)	Normal behavior: download on demand, run simulation
"download"	For use with DryRun() architecture. Downloads data, skips simulation. (In the future, could auto-switch architecture.)
"strict"	Error if any dataset file is missing — never download. For production/cluster runs.

strict mode is implemented as a check in download_dataset:

function check_data_mode(metadata)
    mode = get(ENV, "NUMERICALEARTH_MODE", "")
    if mode == "strict"
        paths = metadata_path(metadata)
        paths = paths isa AbstractVector ? paths : [paths]
        missing_files = filter(!isfile, paths)
        if !isempty(missing_files)
            error("NUMERICALEARTH_MODE=strict but $(length(missing_files)) file(s) missing. " *
                  "First missing: $(first(missing_files))")
        end
    end
end

Workflow: one script, two invocations

# my_simulation.jl — the ONE script
using NumericalEarth, Oceananigans

# User checks ENV to select architecture (initially explicit; auto-switching later)
arch = get(ENV, "NUMERICALEARTH_MODE", "") == "download" ? DryRun() : GPU()

atmosphere = ERA5PrescribedAtmosphere(arch; start_date=..., end_date=..., region=...)
grid = LatitudeLongitudeGrid(arch; size=(1440, 720, 50), ...)
model = EarthSystemModel(grid; atmosphere, ...)
simulation = Simulation(model; ...)
run!(simulation)

# Login node: download all data (fast — minimal grids, no data loading)
NUMERICALEARTH_MODE=download julia --project my_simulation.jl

# Compute node: run with strict guard
NUMERICALEARTH_MODE=strict sbatch run_simulation.sh

Related idea: coarsen grid utility

The grid coarsening logic in DryRun could also be exposed as a standalone utility for testing/development:

# Reduce resolution for quick iteration
test_grid = coarsen(grid, factor=8)         # (1440,720,50) → (180,90,50)
test_grid = coarsen(grid, size=(36, 18, 10)) # explicit target size

This is useful independently of DryRun — for CI tests, development iteration, debugging. The DryRun architecture would use extreme coarsening (to ~4×4×2), while coarsen gives the user control over the factor.

Implementation order

1. DryRun <: AbstractSerialArchitecture with CPU forwarding methods
2. Grid constructor extensions (LatitudeLongitudeGrid, RectilinearGrid, TripolarGrid, ORCAGrid)
3. Field/FieldTimeSeries dispatch for DryRun (download + minimal return)
4. time_step! no-op for DryRun
5. NUMERICALEARTH_MODE=strict guard in download_dataset
6. (Later) coarsen utility
7. (Later) DataManifest.toml read/write
8. (Later) Automatic architecture switching from ENV

[glwagner]

cc @ewquon as well

[awiteck]

How about this approach?
When dir and filename are set, we are in operational (pre-staged) mode. When dir is set but filename is not set, we are in research mode (query to find relevant data, download to disk, load into struct)

[jsheedy]

following @awiteck comment we shouldn't have two separate methods of indicate staged/unstaged data. Since the dir / filename seems required to locate the data, that seems sufficient to indicate existing mode. Does that also open the possibility of staging some data sources but dynamically requesting others?

[glwagner]

Does that also open the possibility of staging some data sources but dynamically requesting others?

This is how the system currently works, since download_dataset will avoid a download if the data is already present. So the purpose of the environment variable is to make the system more robust to errors (if desired).

julia has a command line argument that serves a similar purpose I think: --compiled-modules=strict that @giordano used for some bigger runs we did last year. This argument causes a script to fail if any modules are not precompiled, eg

Enable or disable incremental precompilation of modules. The existing option allows use of existing compiled modules that were previously precompiled, but disallows creation of new precompile files. The strict option is similar, but will error if no precompile file is found.

which can conveniently prevent accidents...

[ewquon]

A few comments

1. I really like the idea of a dry run option. It would also be extremely useful for input validation and debugging. My preference is that no modification to the script be needed — to me, specifying dry_run=true as part of the grid configuration is non-intuitive. Instead, a number of CLI tools have a --dry-run flag which I like.
2. Of course, you would need a REPL equivalent one-liner to turn on the dry-run mode. I'm envisioning this is something that should live in a common module, not sure if that would be NumericalEarth.jl or Oceananigans.jl.
3. I think a dry run mode should also automatically skip postprocessing and plotting -- maybe exit after finishing run!?
4. The defaults for $NUMERICALEARTH_MODE sound reasonable to me.
5. Drawing from my recent experience developing ERA5 downloaders, the automatic metadata download does not necessarily aggregate API requests. So during the dry run mode, perhaps it would make sense to collect all data that would have been downloaded then, if possible, make one combined download_datasets call at the end.
6. We need a metadata struct that handles more than one variable (Aggregate dataset downloads via Vector of Metadata #141).

[glwagner]

My preference is that no modification to the script be needed — to me, specifying dry_run=true as part of the grid configuration is non-intuitive. Instead, a number of CLI tools have a --dry-run flag which I like.

I agree that a non-intrusive method would be amazing! There is a design challenge in how to achieve this. Probably the most robust way would be to build it into the architecture construction --- eg check an environment variable in the constructors for CPU(), GPU(), ReactantState(). It is a little dirty, to return a different type than the constructor indicates. But I think from a practical point of view it can be achieved. The harder question is whether it is a good idea.

A less robust but still possibly viable way is to try to forcibly modify a script, eg finding where architecture is established, and changing it. But that tends to be brittle and has limitations.

So during the dry run mode, perhaps it would make sense to collect all data that would have been downloaded then, if possible, make one combined download_datasets call at the end.

I think this can be achieved by using DryRun() to build a DataManifest.toml during the dry run, and then triggering downloading at the end once all the information has been collected.

We need a metadata struct that handles more than one variable

I believe this is already supported by the fact that Metadata.name can be any value (including Tuple):

NumericalEarth.jl/src/DataWrangling/metadata.jl

Line 41 in 558bf76

name :: S

Perhaps what you mean is to automate support for multivariable? It can be achieved by dataset-specific implementations, but there is no global implementation.

The weird part about multivariable Metadata is that it behaves in a confusing way with Field. Eg would we then want Field(multi_variable_metadata) to return a multi-variable Field? This is perhaps another discussion unrelated to DryRun() though.

Curious what @simone-silvestri and @navidcy think as well

[simone-silvestri]

I am not sure how much this is in scope, but to only download the data another option is a helper script (maybe bash or even just julia) that downloads everything before running. It could be a "preprocessing" script per say, something to be called as:

julia --project dry_run.jl --script_file="my_script.jl"

this will avoid any need to adding to the source code and can parse out all the interactions with metadata and download them.
I guess this is very similar to the DryRun() architecture, but avoids the need to extend Oceananigans functionality?
For 6. names I like the idea of having vectors of metadata to download, the simpler we keep the metadata functionality in my opinon the better.

[glwagner]

How about this approach? When dir and filename are set, we are in operational (pre-staged) mode. When dir is set but filename is not set, we are in research mode (query to find relevant data, download to disk, load into struct)

I think we need a better approach for two reasons. First is that this is implicit --- we aren't changing behavior by saying explicitly "please do this", instead we rely on an "emergent" behavior that is implied by the values of kwargs. This could be fixed by adding a prestaged argument to Metadata though.

More grave is inconvenience and brittleness / failure prone. Imagine we have a typical script with O(10) or even O(100) scattered calls to Metadata throughout. If the user wants to enable prestaged=true, they need to toggle this property for every independent call to Metadata. But this isn't a matter of only finding the calls that the user "owns" / wrote --- Metadata can be called within other functions. One example is regrid_bathymetry, which downloads ETOPO1 data. The user needs to be omnisciently aware of all the places Metadata is called (including, potentially in code that NumericalEarth itself does not own) and pass prestaged=true everywhere. If even one prestaged kwarg is missed, the method fails and a download can potentially be triggered. Failure of the method can only be known by close inspection of logs (presuming that authors of download_dataset are helpful enough to write log messages during downloads). So I think this method is too brittle. What do you think @awiteck ?

[glwagner]

I am not sure how much this is in scope, but to only download the data another option is a helper script (maybe bash or even just julia) that downloads everything before running. It could be a "preprocessing" script per say, something to be called as:

julia --project dry_run.jl --script_file="my_script.jl"

this will avoid any need to adding to the source code and can parse out all the interactions with metadata and download them. I guess this is very similar to the DryRun() architecture, but avoids the need to extend Oceananigans functionality? For 6. names I like the idea of having vectors of metadata to download, the simpler we keep the metadata functionality in my opinon the better.

This is the "forcibly modify a script" method that I mentioned above. The issue is that it is brittle; it is fundamentally impossible to cover all cases (imagine that an architecture or Metadata constructs are hidden in source functions model = build_model()). I think we can put together something that would work for many cases though. It is just not a robust general solution.

[awiteck]

How about this approach? When dir and filename are set, we are in operational (pre-staged) mode. When dir is set but filename is not set, we are in research mode (query to find relevant data, download to disk, load into struct)

I think we need a better approach for two reasons. First is that this is implicit --- we aren't changing behavior by saying explicitly "please do this", instead we rely on an "emergent" behavior that is implied by the values of kwargs. This could be fixed by adding a prestaged argument to Metadata though.

More grave is inconvenience and brittleness / failure prone. Imagine we have a typical script with O(10) or even O(100) scattered calls to Metadata throughout. If the user wants to enable prestaged=true, they need to toggle this property for every independent call to Metadata. But this isn't a matter of only finding the calls that the user "owns" / wrote --- Metadata can be called within other functions. One example is regrid_bathymetry, which downloads ETOPO1 data. The user needs to be omnisciently aware of all the places Metadata is called (including, potentially in code that NumericalEarth itself does not own) and pass prestaged=true everywhere. If even one prestaged kwarg is missed, the method fails and a download can potentially be triggered. Failure of the method can only be known by close inspection of logs (presuming that authors of download_dataset are helpful enough to write log messages during downloads). So I think this method is too brittle. What do you think @awiteck ?

@glwagner yeah I agree with you here. Environment variable approach seems like a decent option here. SSoT for what mode we're in and then the downloader implementations can adapt accordingly. I'm wondering if/how we can simplify the setup we now have with dir, filename, and NUMERICALEARTH_MODE though... seems like there is some overlap in what they are doing (or at least there are now several different cases we need to account for, e.g., dir set, filename unset, NUMERICALEARTH_MODE="strict")