Next steps to create public zarr dataset in mlcast-datasets

[leifdenby]

I am just trawling through the code you wrote @mafdmi (LOL, looking at the commits I wrote some of this and I don't even remember 😆 ) and I will start writing down some questions I have :) With answers to these questions I think we can make a plan for finishing off a version version of the MSGCPP dataset for mlcast-datasets

• Where was the source data downloaded from? The paths currently point to internal DMI paths (https://github.com/mlcast-community/mlcast-dataset-msgcpp/blob/main/msgcpp_to_zarr/source.py#L7). I think you @irenelivia downloaded the source netCDF files from somewhere, right?
  • sorry, I realised I am being an idiot, you mentioned this on Adding the MSGCPP dataset mlcast-datasets#13 , the data can be accessed here: https://msgcpp.knmi.nl/data-access.html
• Rather than using rechunker as you do here https://github.com/mlcast-community/mlcast-dataset-msgcpp/blob/main/msgcpp_to_zarr/zarr_store.py#L47 @mafdmi, did you try simply removing the encoding information? (this is what I realised one can do, and I used it in https://github.com/dmidk/nwp-forecast-zarr-creator/blob/main/zarr_creator/write_zarr.py#L66)
  • this hasn't been tried yet, so we could try that and maybe that way avoid using rechunker (which sometimes requires quite a bit of memory)

[leifdenby]

I just created a script to query the KNMI data portal for all available netCDF files. Just dropping this here so we can integrate it into this repo.

fetch_knmi.py

[mfroelund]

No, I didn't try to remove the encoding information. My approach was to follow your initial code as close as possible, so I didn't spend time on optimizing the code.