Use/promote use of datalad's add-archive-content

[yarikoptic]

I see use/promotion of use of `unzip` for extraction from zip files eg of results

babs/generate_bidsapp_runscript.py:        commands to unzip input datasets
babs/generate_bidsapp_runscript.py-    """
babs/generate_bidsapp_runscript.py-    if not any(ds['is_zipped'] for ds in input_datasets):
babs/generate_bidsapp_runscript.py-        return ''
babs/generate_bidsapp_runscript.py-
babs/generate_bidsapp_runscript.py-    env = Environment(
babs/generate_bidsapp_runscript.py-        loader=PackageLoader('babs', 'templates'),
--
docs/after_jobs.rst:And you can unzip it via::
docs/after_jobs.rst-
docs/after_jobs.rst:    unzip zip_file_name.zip
docs/after_jobs.rst-
docs/after_jobs.rst-What if my input data changes?
docs/after_jobs.rst-==============================
docs/after_jobs.rst-
docs/after_jobs.rst-If your input data changes, you can use ``babs update-input-data`` to update the job status dataframe.
docs/after_jobs.rst-See :doc:`babs-update-input-data` for details.
--
docs/developer_how_to_test.rst:Finally, check if you can successfully merge the results + get the zip file content + unzip it.
--
docs/overview.rst:After you clone and unzip the results from the BABS project, they are ready to be consumed!
docs/overview.rst-Details about results merging and consuming are on this page: :doc:`after_jobs`.
docs/overview.rst-
docs/overview.rst-========
docs/overview.rst-See also
docs/overview.rst-========
docs/overview.rst-
--
docs/walkthrough.rst:Then use ``unzip -l`` to list the content of the zip file:
docs/walkthrough.rst-
docs/walkthrough.rst-..  code-block:: console
docs/walkthrough.rst-
docs/walkthrough.rst-    $ datalad get sub-0001_ses-01_fmriprep_anat-25-0-0.zip
docs/walkthrough.rst:    $ unzip -l sub-0001_ses-01_fmriprep_anat-25-0-0.zip
docs/walkthrough.rst-

but I wonder if instead use of https://docs.datalad.org/en/stable/generated/man/datalad-add-archive-content.html (datalad add-archive-content) should be promoted to not just extract but commit to git/git-annex with linkage to those archives so content could be annex get'ed from them later.

In case of multiple .zip, i think we take only 1 at a time ATM, I came up with this abomination but we should likely to address it properly in datalad and @mih probably did that in one of this "new datalad" versions:

datalad run -m "Extracting all .zip files" --input '*.zip' 'for f in *.zip; do datalad add-archive-content -D --allow-dirty --no-commit --strip-leading-dirs --leading-dirs-depth 1 $f; done'

[mattcieslak]

We had tried using add-archive-content a few years ago and it was very unstable/didn't work very well. Have you had luck using it at scale?

[yarikoptic]

IIRC that, or functionality under, what was used to keep the original datalad archive of openfmri https://datasets.datalad.org/?dir=/openfmri and CRCNS https://datasets.datalad.org/?dir=/crcns going and likely others...

What were your particular concerns and issues (if filed)? if none -- that might have all was a bad dream? ;)