Status of this project

[ericonr]

Hi @ulrikpedersen! I hope you are doing well.

I came across this project, and would like to understand its current status.

I found https://epaper.kek.jp/ICALEPCS2013/papers/wecoba07.pdf , which documents some tests on distributed file systems.

Was the final conclusion that Virtual Datasets (or other multiple-file alternatives) should be pursued and phdf5write abandoned, then?

Did you perform any further testing with this project? Do you know/feel the reported HDF5+MPI limitations you faced are still present?

At least support for compression with parallel HDF5 seems to have been implemented: https://www.hdfgroup.org/2022/03/04/parallel-compression-improvements-in-hdf5-1-13-1/

[ulrikpedersen]

Hi @ericonr

The current status as you can probably tell from the commit history is that is has been dormant for 12 years 😀 It has not been used at any facility in that time since the last edits.

Yes, you found the right paper, and yes the conclusion was that writing a single file per process was a much better option both in terms of performance but also in terms of stability. Our requirement from beamline was to be able to collect continuously 1M frames without any dropped.

Soon after this paper, we dropped phdf5 entirely and raised the projects to develop both SWMR and Virtual Dataset with the HDF5 group. You can follow the trail of developments through ICALEPCS proceedings: the following conference in 2015 we published this.

For scalable/modular detector systems that had to perform at > 10gbps or with multiple hosts/NICs, we then started to develop the Odin framework for DAQ - and this is still going strong at Diamond, STFC and other facilities. Odin handles the high speed data capture to HDF5 (using SWMR and optionally Virtual Dataset) but can be controlled from an EPICS areaDetector driver so that it integrates with the beamline control systems.

I have not done any further work with HDF5 and MPI-IO since then. While I am sure that improvements have been made in the meantime, I am convinced that this is not a suitable mechanism to use for live DAQ systems: This is designed for achieving high performance in aggregate numbers across 10s of thousands of nodes - not 10s of nodes. For lower numbers it performed quite badly - but it was also brittle and unstable and quite unsuitable to deal with the continuous real-time stream of data from a DAQ system. Some nodes would progress (i.e. write to disk) faster than others and when nodes were much out of synch with each other they would begin to slow down, despite there being no need for communication between them. Deadlocks and crashes were frequent as well.

I am not currently involved with Odin developments but I know there is a capable team of engineers working on this at STFC and Diamond. If you have a need to build a DAQ system for a new, high-performance, and scalable/modular detector systems, then I would thoroughly recommend checking out this project and reach out to the team that maintains it.

[ericonr]

Hi Ulrik, thank you for the quick answer!

You can follow the trail of developments through ICALEPCS proceedings: the following conference in 2015 we published this.

Thanks for the reference, it's a really nice introduction to those features :)

We were aware of Odin, but hadn't realized it had come about as a followup to work on parallel HDF5. It's one of the software frameworks we are evaluating for our new detectors!

Some nodes would progress (i.e. write to disk) faster than others and when nodes were much out of synch with each other they would begin to slow down, despite there being no need for communication between them. Deadlocks and crashes were frequent as well.

That's unfortunate. Using MPI to be able to store data into a single file seemed quite practical and elegant, but I suppose there's a reason none of the current frameworks I found made use of it. Thank you for sharing your experience.

I am not currently involved with Odin developments but I know there is a capable team of engineers working on this at STFC and Diamond. If you have a need to build a DAQ system for a new, high-performance, and scalable/modular detector systems, then I would thoroughly recommend checking out this project and reach out to the team that maintains it.

We've talked to Gary Yendell, he has been very helpful so far!