Practical Strategies for YODA in the Wild

[Servinjesus1]

One incredible benefit to YODA (or in general annex-enabled repos) is that the resulting repo is very lightweight and thus the overhead of importing an entire repo as a submodule (code, inputs, outputs, environments, and all) is quite small if one doesn't also import all the annexed file contents.

However, for large codebases or involved analyses, the git repo might get quite hefty. In that case, it might be advisable to save data outputs to a submodule itself - a 'raw' style repo that just annexes data files with no processing or output layers. These 'raw' inputs then are maximally lightweight (just a git-tree of symlinks).

Granted, taking this to the extreme would result in lots of tiny repos floating around so I've considered solutions like grouping a bunch of 'raw' repos by branches into one repo. This of course introduces questions of maintaining provenance and readability (having several branches to checkout to navigate files).

Has anyone run into the issue where large datasets are a bit of a pain to import into subsequent stages of the analysis process? This would happen if the git repo itself becomes quite large (hundreds of commits or thousands of files). How do you store your repos and file contents and what convention might you suggest for storing raw repos so you don't get too inundated with raw repos?

For me, I use github for repos, with dropbox (rclone special remote) and a research server (bare annex-enabled repo) for file content storage. I've considered storing raw repos on github under dot-prefixed names (so if the dataset is at Collab/Proj-Phase.git, the raw repo(s) for this set are at Collab/.Proj-Phase.git with plural raws saved as branches within this repo). I also quite like git-annex's new git-remote-annex which allows the storage of git guts in a special remote with push/pull support because it also allows storing an arbitrary number of separate repos in the same place (alongside the content itself). Note it doesn't yet work with rclone.

For readability I am also considering using exporttree annexes - especially to dropbox - so a majority of file contents is in human-readable named format. When it comes to versioning file contents I try to separately name different variants of files and only update annexed files if the old version should no longer be referenced.

These are some of the ways I use YODA in the wild - whether writing my dissertation or performing analysis - so provenance is maintained, duplication of random files floating around is minimized, and file contents are still generally discoverable during the aggregation phase of e.g. writing a report. I would love to hear more about others' strategies in particular when datasets get large or how to use YODA when splitting processing/analysis into subdatasets for e.g. parallel processing.

[yarikoptic]

Hi @Servinjesus1 , thanks for reaching out! Lots of good thinking and information in your post for others to potentially digest. A quick reply with some "bullet pointed" items which relate to your questions, even if not creating a complete cohesive story, since there are multiple! ;)

• check out "FAIRly big..." paper from @mih @adswa et al : https://www.nature.com/articles/s41597-022-01163-2 which goes through architecturing setup which is also described in https://handbook.datalad.org/en/latest/beyond_basics/101-170-dataladrun.html#processing-fairly-and-in-parallel-general-workflow and other referenced there sections. The overall idea -- separate "data units" for processing go into separate subdatasets and RIA stores identifying datasets by their UUIDs would provide you the "backend" storage for those individual subdatasets
• in https://dandiarchive.org we are accepting Zarr filesets where each one could have hundreds of thousands of files inside. Each .zarr, identifier by its UUID, becomes a separate DataLad dataset, and shared on a dedicated to them github organization: https://github.com/dandizarrs/ whenever dandisets (potentially) containing them are at https://github.com/dandisets with an ultimate "super-dataset" across them at https://github.com/dandi/dandisets . An example of a dandiset with many zarrs: https://github.com/dandisets/000108/ (its description "says it all": 332739854 files, 374.4 TB -- that's across all subdatasets). Hence transfer/processing etc could be done per individual "zarr"
• in our field we are using/developing BIDS data standard: https://bids.neuroimaging.io/ where data for each subject goes into a separate folder, and entire dataset is composition of such folders.
  • Whenever there is too many subjects, we recommend/do - make each subject into a subdataset. Also potentially providing "filtered views" with only specific subsets of files present for a specific target "data modality", e.g.
    • https://github.com/datalad-datasets/human-connectome-project-openaccess contains submodules (RIA store with URL in https://github.com/datalad-datasets/human-connectome-project-openaccess/blob/master/.datalad/config#L3), while e.g. https://github.com/datalad-datasets/hcp-functional-connectivity is a single "derived" dataset providing all subjects data but only for a particular data modality etc
    • we are approaching "formalization" of more modular organization of BIDS datasets from smaller constituents in so we could "ship" a dedicated subdataset for processing independently from the rest while retaining it as a valid BIDS dataset: Allow composition of a BIDS dataset (dataset level) from smaller (subj or subj/ses) level bids-standard/bids-2-devel#59 . Check references therein
      • related approaches are placing each subject/session into a separate dataset and then "merging" them as desired
• just at a conference I am coming from (SfN), saw a poster (unfortunately was not open for photo and can't search up the author here) where they utilize git branches per data acquisition/processes, thus also making it possible to operate on separate units without transferring/checking them out all at once. (relates to the above method since branch is just potentially a manifestation of a tree on a disc as in above layouts)

... bus arrived ... may be more later ... stay in touch, let's keep dialog going