Found a few documentation bugs

[ummaraali2]

While testing the README examples, I found a couple issues in the
Using Jupyter Notebook or a Script section:

README.md:

1. Missing cases in import statement (later code uses cases.load_ewb_events_yaml_into_case_collection())
2. KerchunkForecast example missing required name parameter

Quickstart notebook (docs/notebooks/quickstart.ipynb):

• References from extremeweatherbench import config, events, evaluate but config module doesn't exist and events/__init__.py is empty

I have working fixes for the README. For the notebook, not sure if you're planning a rewrite or if I should update it to match current API.

Happy to submit PRs for whichever would be helpful! Or would like to know if I may have to learn something regarding this.

[aaTman]

Thank you for the catches! You are definitely correct and they need to be fixed. You are more than welcome to submit the changes as a PR.

One of the top items we're prioritizing currently is a review and refresh of the existing documentation based around the new API. If you happen to run into any pain points or issues that could use more clarity feel free to let us know.

[aaTman]

Hmm, most of the heat wave events should be covered in the FCNv2 GFS initialized data from CIRA. Did you set variable_mapping in the KerchunkForecast initialization to make sure the CIRA variables are renamed? We have a built-in mapping for CIRA data in inputs (inputs.CIRA_metadata_variable_mapping) as a convenience, considering how often we reference those datasets.

[ummaraali2]

Yes, I used variable_mapping=inputs.CIRA_metadata_variable_mapping. When I checked the time range:

ds = fcnv2_forecast.open_and_maybe_preprocess_data_from_source()
print(ds.time.values.min(), ds.time.values.max())
# 2020-09-30 to 2020-10-10 (only 11 days)

So FOUR_v200_GFS.parq (found in readme) only covers ~11 days - none of the heat wave cases fall in that window.

Also, I found docs/examples/applied_heatwave.py which uses HRES data instead:

source="gs://weatherbench2/datasets/hres/2016-2022-0012-1440x721.zarr"

Running with HRES gave me 697 rows of results for 2016-2022 cases. Might be worth noting this in the docs!

[aaTman]

Ah! I see the issue and it is my fault. The way virtualizarr/kerchunk is set up made it so this data needs a preprocessing function which I failed to include in this snippet. I'll modify the readme now for that and to include the required name argument but for brevity, here is the relevant preprocessing function that will address this:

def preprocess_bb_cira_forecast_dataset(ds: xr.Dataset) -> xr.Dataset:
    """Preprocess CIRA kerchunk (parquet) data in the ExtremeWeatherBench bucket.

    A preprocess function that renames the time coordinate to lead_time,
    creates a valid_time coordinate, and sets the lead time range and resolution not
    present in the original dataset.

    Args:
        ds: The forecast dataset to rename.

    Returns:
        The renamed forecast dataset.
    """
    ds = ds.rename({"time": "lead_time"})

    # The evaluation configuration is used to set the lead time range and resolution.
    ds["lead_time"] = np.array(
        [i for i in range(0, 241, 6)], dtype="timedelta64[h]"
    ).astype("timedelta64[ns]")

    return ds

and applying it to the code:

fcnv2_forecast = inputs.KerchunkForecast(
    name='fcnv2_forecast',
    source=forecast_dir, # source path
    variables=["surface_air_temperature"], # variables to use in the evaluation
    variable_mapping=inputs.CIRA_metadata_variable_mapping, # mapping to use for variables in forecast dataset to EWB variable names
    storage_options={"remote_protocol": "s3", "remote_options": {"anon": True}}, # storage options for access
    preprocess=preprocess_bb_cira_forecast_dataset,
)

[ummaraali2]

Thanks! Will try this. Also, been wondering about - if scaling to more models, would each data source potentially need both a mapping AND a preprocessing function?

[aaTman]

It depends on the orientation of the dimensions. If the data is already set up with dimensions of lead time, initialization time, and one or more spatial dimensions, PLUS the variables to evaluate already exist... no preprocessing is needed.

In other words, preprocessing right now has two use cases: prepping the dimensions of the forecast data and calculations of simpler variables that don't trigger computations (loads into memory like complex reductions). Ideally, a user should have already done these steps outside of EWB but there are clearly cases that isn't realistic.

As for mapping, most likely... as most forecast models have distinct names for their outputs, e.g. t2m, 2m_temperature surface_temperature, t2, temperature_at_2_meters_above_surface, etc. It's an easy (in my opinion) way to manage what is likely to be a common "problem" with meteorological datasets!

[amymcgovern]

I'm running all the heat waves regularly. I have a bunch more example scripts in the repo I am using for our paper.

One note, the CIRA archive is pretty incomplete on IFS initialization. I ended up using GFS only because of all the missing cases.