Support setting DSP output column level HDF5 settings

[gipert]

Would be needed if we want to use custom compression settings or any other HDF5 dataset option. We could actually have a new JSON field listing attributes that should be attached to LGDOs, e.g.

"wf_bl": {
    "function": "bl_subtract",
    "module": "dspeed.processors",
    "args": ["waveform", "baseline", "wf_bl"],
    "unit": "ADC",
    "lgdo_attrs": {
        "0vbb": "I want to believe!",
        "hdf5_settings": {
            "compression": "gzip",
            "shuffle": true
        }
    }
}

[iguinn]

I added this feature, but it doesn't work for this:

        "hdf5_settings": {
            "compression": "gzip",
            "shuffle": true
        }

giving TypeError: Object dtype dtype('O') has no native HDF5 equivalent. Is this how you had intended it to be used for setting the compression options, or am I missing something?

However, it does work for basic strings. I tried adding a "description" attribute, and one really nice thing is that it can be displayed using the -a option for lh5ls!

├── ch1027200 · HDF5 group 
│   └── dsp · table{wf_presum,waveform_windowed} 
│       ├── waveform_windowed · table{t0,dt,values} ── {'description': 'Waveform windowed from 42 us to 64.4 us'}
│       │   ├── dt · array<1>{real} ── {'units': 'ns'}
│       │   ├── t0 · array<1>{real} ── {'units': 'ns'}
│       │   └── values · array_of_equalsized_arrays<1,1>{real} 
│       └── wf_presum · table{t0,dt,values} ── {'description': 'Waveform presummed by a factor of 6'}
│           ├── dt · array<1>{real} ── {'units': 'ns'}
│           ├── t0 · array<1>{real} ── {'units': 'ns'}
│           └── values · array_of_equalsized_arrays<1,1>{real} 

I think this would be a great feature to take advantage of to make our analysis self-documenting.

[gipert]

giving TypeError: Object dtype dtype('O') has no native HDF5 equivalent. Is this how you had intended it to be used for setting the compression options, or am I missing something?

Uhm it's supposed to work, see:

https://legend-pydataobj.readthedocs.io/en/latest/api/lgdo.lh5.html#lgdo.lh5.store.LH5Store.write

let me have a look...

[gipert]

I implemented exactly the same thing here:

legend-exp/pygama#543

and it works... did you check your legend-pydataobj version?

[iguinn]

Oh I think the problem is that it doesn't work on WaveformTables, but it does on other objects. I'm guessing it doesn't work on composite LGDO objects?

[gipert]

Indeed!

[iguinn]

How do we want to handle waveform compression then? Do we want to have the LGDO WaveformArray object handle propagating the options to the table values, or do we want to have the processing_chain handle it?

[gipert]

Good point... I guess there is no clean way to do this. I think we need something similar to what I have implemented for the daq2lh5 configuration:

https://legend-daq2lh5.readthedocs.io/en/latest/api/daq2lh5.html#daq2lh5.data_decoder.DataDecoder

i.e.

"wf_bl": {
    "function": "bl_subtract",
    "module": "dspeed.processors",
    "args": ["waveform", "baseline", "wf_bl"],
    "unit": "ADC",
    "waveform_compression": {
        "values": "RadwareSigcompress(codec_shift=-32768)",
        "t0": "gzip"
    },
    "lgdo_attrs": {
        "0vbb": "I want to believe!",
        "hdf5_settings": {
            "compression": "gzip",
            "shuffle": true
        }
    }
}

or similar. This extra waveform_compression field would only work for WaveformTable outputs, obviously. Not sure how to specify extra HDF5 settings for the WaveformTable fields though... happy to know if you have better ideas.

For dev, this is useful to convert waveform decoder strings to Python objects: https://legend-pydataobj.readthedocs.io/en/stable/api/lgdo.compression.html#lgdo.compression.utils.str2wfcodec

[iguinn]

#44 has been accepted; however, this is only a partial solution to this issue so I am leaving it open