Adding CDL Parser/`open_cdl`?

[nbren12]

Is your feature request related to a problem?

No.

Describe the solution you'd like

It would be nice to load/generate xarray datasets from Common Data Language (CDL) descriptions. CDL is a DSL that that defines a netCDF dataset, and is quite nice for testing. We use it to build mock datasets for e.g. integration testing of plotting routines/complex data analysis etc. CDL provides a concise format for storing the schema of this data. This schema can be used for validation or generation (using the CLI ncgen).

CDL is basically the format produced by xarray.Dataset.info. It looks like this:

  netcdf example {   // example of CDL notation
  dimensions:
	  lon = 3 ;
	  lat = 8 ;
  variables:
	  float rh(lon, lat) ;
		  rh:units = "percent" ;
		  rh:long_name = "Relative humidity" ;
  // global attributes
	  :title = "Simple example, lacks some conventions" ;
  data:
 /// optional ...ncgen will still build 
   rh =
    2, 3, 5, 7, 11, 13, 17, 19,
    23, 29, 31, 37, 41, 43, 47, 53,
    59, 61, 67, 71, 73, 79, 83, 89 ;
  }

I wrote a small pure python parser for CDL last night and it seems work! There are similar projects on github. Sadly, these projects seem to be abandoned so it would be nice to attach to an effort like xarray.

Describe alternatives you've considered

Some kind of schema object that can be used to validate or generate an xarray Dataset, but does not contain any data.

Additional context

No response

[kmuehlbauer]

@nbren12 The other way round would be useful too. Aren't there xarray extension packages around where this would fit into?

[nbren12]

Aren't there xarray extension packages around where this would fit into?

I'm not sure. Any suggestions? Just wondering if xarray has left the door open to this kind of contribution since it

1. already supports other i/o backends
2. creates CDL using ds.info().

[dcherian]

@nbren12 See https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html for adding a new backend. That way we could have

xr.open_dataset('schema.cdl', engine="cdl")

[kmuehlbauer]

2. creates CDL using ds.info().

Great, this somehow went past me.

[nbren12]

To be fair, ds.info is not 100% CDL, but it's darn close.

[jhamman]

To be fair, ds.info is not 100% CDL, but it's darn close.

I think making ds.info CDL compliant would be a great feature addition.

Describe alternatives you've considered

Some kind of schema object that can be used to validate or generate an xarray Dataset, but does not contain any data.

You may be interested in xarray-schema then. We're actively working on / using this project and would be more than happy to think about how a cdl-like schema fits in there.

[nbren12]

@jhamman We have a similar schema package https://github.com/ai2cm/fv3net/tree/master/external/synth, cool to see you confronting the same challenges and advertising your solutions more broadly. One problem we had is that our schema objects ended up being quite verbose: https://github.com/ai2cm/fv3net/blob/master/external/loaders/tests/test__batch/one_step_zarr_schema.json. CDL is a lot more concise.

[brendan-m-murphy]

You can do something like:

cdl = netCDF4.Dataset.fromcdl("example.cdl")
ds = xr.open_dataset(xr.backends.NetCDF4DataStore(cdl))

This creates example.nc in the background, which I don't really want to happen. Also you can't change fill values, so in my case, I ended up setting the value of an empty time dimension to 0.0. You could also avoid problems here by using decode_time=False in the open_dataset call.

[nbren12]

I decided to make a small package for this: https://github.com/nbren12/xarray_cdl

This creates example.nc in the background, which I don't really want to happen.

@brendan-m-murphy the parser in my package is written in pure python, and doesn't use the netCDF library.

I think making ds.info CDL compliant would be a great feature addition.

@jhamman would this feature still make sense? One issue is that CDL doesn't support all np.dtypes---np.datetime64 for example. So this conversion would be lossy.

[brendan-m-murphy]

For datetimes, if the attributes are cf units for time then there is a parser in xarray that will reconstruct the datetimes from integer (float?) values.

[VeckoTheGecko]

CDL is a DSL that that defines a netCDF dataset, and is quite nice for testing

We've found something that's nice. It's not CDL - but I've found the ouput of ds.to_dict(data=False) to be very rich allowing you to recreate datasets from others without having to share all the data. Only problem from my POV is that coordinates data isn't included (and the fact that this output format isn't standard AFAIK)

Our contributor docs mentioning how to "share datasets" like this: https://github.com/Parcels-code/Parcels/blob/9f3f77d4178ab443bdc52b475deb92d9e002a6b8/docs/development/posting-issues.md?plain=1#L18-L68
Our tooling to support this: https://github.com/Parcels-code/Parcels/blob/9f3f77d4178ab443bdc52b475deb92d9e002a6b8/src/parcels/_datasets/utils.py#L189-L217

xref #2656

EDIT: Coordinate array data was quite important for us so we opted for Parcels-code/Parcels#2570 instead