archivist support of other archive formats

[bpinsard]

Hi there,

We are using the older archive special remote, and I am thinking about migrating to archivist.
However, we are currently using 7zip with solid mode off with the idea to enable "fast" random access in the archive, even with compression, though with a penalty with solid-mode=off.
My understanding is that archivist would be able to do such random access, while the archive special remote was not (fully un-archive in cache)
Would you be willing to accept a PR for new p7zip backend for archivist?

That question comes from stumbling on https://hub.trr379.de/q02/dicom-utilities/src/branch/trunk/dataladify_dicom_dataset.py ( @mslw can't comment on the repo).
How scalable is that? We have a similar approach but with the older archive backend, and we hit perf issues with a GE scanner that generates 100k+ dicoms per session(single slice dicoms, multi-echo fMRI or DWI, mag+phase), while it works with the usual volumes of old-mosaic or enhanced Siemens dicoms.

Thanks!

[mslw]

We are using the older archive special remote, and I am thinking about migrating to archivist. However, we are currently using 7zip with solid mode off with the idea to enable "fast" random access in the archive, even with compression, though with a penalty with solid-mode=off. My understanding is that archivist would be able to do such random access, while the archive special remote was not (fully un-archive in cache) Would you be willing to accept a PR for new p7zip backend for archivist?

In general, I don't see why not. The more the merrier.

However, AFAIK, the archivist special remote is based on datalad_next/archive_operations, which uses tarfile & zipfile from Python's standard library. I don't think there is built-in 7-zip support, so the only potential point of contention might be additional dependency footprint - @mih might have a more specific opinion on that.

That question comes from stumbling on https://hub.trr379.de/q02/dicom-utilities/src/branch/trunk/dataladify_dicom_dataset.py ( @mslw can't comment on the repo). How scalable is that? We have a similar approach but with the older archive backend, and we hit perf issues with a GE scanner that generates 100k+ dicoms per session(single slice dicoms, multi-echo fMRI or DWI, mag+phase), while it works with the usual volumes of old-mosaic or enhanced Siemens dicoms.

You found me anyway (with delay) 😉

That script which you linked has two objectives: create a tarball in a bitwise reproducible manner (which requires setting timestamps - we use timestamps from DICOM header) and keep the DICOMs in the worktree with some metadata attached (archivist URL for access, obviously, but also some basic sequence info as git-annex metadata, which can be used with git annex view to reorder the worktree).

I only ever saw Siemens DICOMs, so I don't know where the limits are. I also don't have a good benchmark - for acqusitions which I would consider "large" there is some waiting, but IIRC more time is spent reading information from the DICOM header than adding archivist URLs. However, I imagine that when you are going from 10k+ to 100k+ files you might be pushing the performance limits with Git repo as well, maybe?

I imagine at some point it might make sense to only index the archive in the dataset instead of the individual files (while neat, the views aren't very practical as most people would convert the entire acquisition most of the time), but it's hard for me to tell where that point is, exactly.