Notebook demonstrating direct S3 access of the HYCOM reanalysis data on AWS Open Data #1
Description
@emmcdonald and others, I created references for the HYCOM Reanalysis data on AWS Open Data that allows you to use xarray and the zarr library to access the files as a single virtual dataset.
Data access example notebook
(the python environment to run it requires only standard conda-forge packages)
For these NetCDF 64-bit offset files, we created virtual "chunks" which cover the entire spatial domain, but with one vertical layer (and one time step, of course). If you are curious about the details of what we did, here is the [Reference generation notebook] (https://nbviewer.org/gist/rsignell/a97619c204f5d27c99e09a4426f7aa97). Note that to run this you currently require the latest kerchunk from main branch on github.
Please check out the access notebook and if you approve, I'd be happy to submit a PR to the dataset page at https://registry.opendata.aws/hycom-gofs-3pt1-reanalysis/. We could also add some other examples, like subsetting, or taking the mean over some time range, etc. We would probably want to move the references from the Open Storage Network bucket they are currently on to the open data bucket so they can be on a more reliable long-term bucket along with the data files. We might also want to fix my spelling of HyCOM=>HYCOM. oops. 😄
I'd be happy to have a short web meeting to demo/discuss all this, also, if you are interested.