How to use with MPI jobs?

[bilke]

Imagine I have a rule which should run mytool via MPI on 8 processes.

Locally I would start this with e.g.: mpirun -np 8 mytool.

On a slurm cluster I would write a submit script and then inside the script start with srun, e.g.:

#!/bin/bash
#SBATCH --job-name=myjob1
#SBATCH -n 20
...
srun mytool

For MPI and slurm it is crucial to start the parallelized application (mytool) via srun, see also https://slurm.schedmd.com/mpi_guide.html.

---

I assumed that when using this profile to submit jobs to a slurm cluster using Snakemake the following would happen:

• A similar submit script like above is created
• The submit script is submitted via sbatch

But instead Snakemake does the following:

• It creates a slurm job
• Inside the job snakemake is run again with lots of parameters to ensure that only the current rule is run (basically the contents of {exec_job} in https://github.com/Snakemake-Profiles/slurm/blob/master/%7B%7Bcookiecutter.profile_name%7D%7D/slurm-jobscript.sh)

---

So my questions are:

• Is it possible to modify this profile to get a similar behavior I originally assumed (directly executing srun inside the job)?
• Have others successfully run MPI parallelized jobs on multiple nodes with Snakemake?

Explicitly pinging @jdblischak because you may have experience on this topic as well.

cc: @TobiasMeisel

[jdblischak]

@bilke I'm not an MPI expert, but I agree it would be nice to support this use case if possible

Locally I would start this with e.g.: mpirun -np 8 mytool.

Do you have an example command that I could use for testing? Preferably using a tool available from conda-forge or bioconda

For MPI and slurm it is crucial to start the parallelized application (mytool) via srun, see also https://slurm.schedmd.com/mpi_guide.html.

Have you tried only using sbatch? As far as I can tell, the HPC docs at my company only use sbatch for MPI jobs (ie they put the mpirun command directly into a shell script with #SBATCH directives)