Should we bring the DataLad Slurm extension to DataLad Next?

[knuedd]

We created a DataLad plugin for Slurm for so that DataLad is better suited for HPC (High Performance Computing) with the Slurm batch processing system. The DataLad plugin is available at https://github.com/knuedd/datalad-slurm and the design description as well as the evaluation can be found in the paper preprint at https://arxiv.org/abs/2505.06558.

What would you recommend as the next step? Should we bring this extension into DataLad-Next? Or what would be the prerequisites? What should we present or discuss before?

Best, Andreas

[mih]

I think having SLURM-focused functionality in an extension package is already quite optimal.

I see that datalad-slurm does not add external dependencies, but testing it complicates the rest of datalad-next testing quite a bit. It should (remain) more efficient to ensure proper functioning of datalad-slurm when its testing can focus on the needs of a SLURM environment.

What would be your motivation to move the code to datalad-next?

[knuedd]

Sure, it should work fine as a separate extension package and I see that it is a rather specific subgroup of users who would want to use it. If the functionality would be available in the distributed version it would be easier for those w.r.t. installation etc. But I understand the extra complications for automated tests that datalad-slurm would cause.

The step to datalad-next was my understanding what the next step should be to get it into standard datalad eventually. I'm good with keeping it a separate extension for now.

There would be some follow-up questions, though.

1. How to streamline fixes from https://github.com/datalad/datalad-extensions go to my extension? For BF: use setuptools.errors.OptionError instead of now removed import of distutils.DistutilsOptionError; stop using python 3.8 in CI datalad-extension-template#98 it was necessary the other day ...
2. Could we place some pointers in the official DataLad documentation or in the DataLad handbook so that HPC users can find the extension more easily?

... I guess I bring those questions to the DataLad Office hour some time.

[mih]

re installation: you could depend directly on datalad or datalad-next in datalad-slurm. The extension package would become the main installation target for its users. This gives you more leverage re the runtime environment.

For example, there is now a git-annex wheel available on PyPi (https://pypi.org/project/git-annex), which you could decide to depend on (make the custom installation of git-annex obsolete). Eventually, datalad might do that too, but with the extension being the main target, you can decide at your own pace.

[mih]

re documentation pointers: yes, sure. The handbook has a list that your could propose an addition to https://handbook.datalad.org/en/latest/extension_pkgs.html