Feature Request: Make zipping optional

[asmacdo]

Summary

Currently BABS always zips outputs. For my workflows, raw BIDS-derivative output would be preferable because it enables cleaner integration into BIDS-study layouts with better provenance tracking.

In particular, zipping works fine for individual usage where some manual intervention is acceptable, but for larger automation workflows with many datasets, it would be ideal to produce the outputs into their final place.

BIDS-study output structure goal

See BIDS common principles and PR #1741 for background on study-level dataset organization.

my-study/
  sourcedata/raw/               # BIDS raw dataset (subdataset)
  code/
    containers/                 # containers should be considered part of code
  derivatives/
    mriqc/                      # MRIQC outputs (subdataset)
      sourcedata/raw/           # reference to input (YODA)
      code/                     # processing scripts (provenance)
      sub-01/
      sub-02/
      dataset_description.json  # single dataset_description instead of 1 from each subject
    fmriprep/                   # fMRIPrep outputs (subdataset)
      sourcedata/raw/
      code/
      sub-01/
      sub-02/
      dataset_description.json

With raw outputs (no zip), after babs merge you could clone the output_ria directly into the BIDS-study:

datalad clone ria+file://output_ria#~data derivatives/mriqc

This gets very close to the desired structure - the clone is the final derivative, ready to use, with full provenance intact.

(Note: BABS's default input path is inputs/data, but this is overridable via path_in_babs config to use sourcedata/raw for BIDS-study compliance.)

Problems with current zip-only approach

• Provenance obscured: Unzipping with datalad run records "unzip" as provenance rather than the original BIDS app command. The true provenance isn't lost, but it's obscured.
• Duplicate large files: After unzipping, git-annex has both the zips and the raw files. These can be dropped, but it's an extra step that is easily forgotten. Minor but potentially wasteful.
• Unzip conflicts: Each per-subject zip contains shared files like dataset_description.json and .bidsignore. When unzipping, these conflict. They're probably identical (same container version = same output), but with raw outputs stored in git (not zips), merge would catch any surprises.

For raw outputs to work

For per-subject branches to merge cleanly, JSON and TSV files should be stored in git (not annex) - either via text2git config or .gitattributes. This allows git's octopus merge to handle identical files, and surface conflicts if they unexpectedly differ.

Suggested config option

Ideally the user could choose:

• Raw outputs committed (no zip)
• Zipped outputs (current behavior)

Implementation approaches

Least invasive: Modify existing script generation template to optionally include the zip step. The generated *_zip.sh script already handles both execution and zipping - it could conditionally skip the zip.

With containers-run (see #328): Separating containers-run from zip enables two explicit commits - one for the BIDS app, one for zipping. This gives cleaner provenance and makes the zip step trivially optional, but is a larger change.

[yarikoptic]

Somewhat related here is

• Use/promote use of datalad's add-archive-content #343
  and I really wonder what BABS folks think about this particular issue, since in a case where we do not care about even having RIA, and want to operate on a shared NFS folder to benefit from ephemeral (worktree) clones (see Support git worktree #338) -- I see no point in forcing people to use RIA stores!

I thought we could just may be make analysis/ folder to be that bids-derivative dataset (e.g. derivitives/fmriprep-1.0.0) to start with and place temporary for us RIAs somewhere else and then just pull and merge all the branches there, but apparently even the name analysis is hardcoded in https://github.com/PennLINC/babs/blob/HEAD/babs/base.py#L105

        self.analysis_path = op.join(self.project_root, 'analysis')
        self._analysis_datalad_handle = None

        self.config_path = op.join(self.analysis_path, 'code/babs_proj_config.yaml')

        self.input_ria_path = op.join(self.project_root, 'input_ria')
        self.output_ria_path = op.join(self.project_root, 'output_ria')

        self.input_ria_url = 'ria+file://' + self.input_ria_path
        self.output_ria_url = 'ria+file://' + self.output_ria_path

Dear BABS developers, could you please follow up on this issue with your thoughts? Thank you in advance!

[asmacdo]

Also could be reasonable to keep rias in analysis and gitignore IMO

[tien-tong]

Following up on on some issues raised here. In the current BABS project organization: both input_ria and output_ria are siblings of analysis, so the RIAs are not optional and analysis is not equivalent to derivatives/BIDS_app. The motivation for this organization is this paper: https://www.nature.com/articles/s41597-022-01163-2, @mattcieslak can explain this better than I do. We can definitely discuss more when we meet.

[yarikoptic]

analysis is not equivalent to derivatives/BIDS_app

how do you see it different, or in other words what is its unique function in life?

The motivation for this organization is this paper: https://www.nature.com/articles/s41597-022-01163-2, ...

yeap. And one of the motivations there was inspired by condor being able to work across systems even without shared filesystem. Hence to a degree, in addition to inodes consumption minimization, and avoiding conflicts upon pushes to the same remote, inspired all the push/pull via RIAs.

But when the analysis is a regular git/git-annex repo to push/pull to, and dataset scale is not forbidding keeping all files extracted, IMHO RIA's are not strictly necessary. (although some issues still remain with parallel push/annex-copy'ing ATM and might need to be addressed first or just serialized via locking for a workaround as we do have shared filesystem to make it possible)