Running multiple pipelines with nf-scil on clusters on a lot of subjects

[AlexVCaron]

Running multiple pipelines, or a long sequence of processes, will create a lot of intermediary files. This will not work on clusters, where we already have problems (tractflow+rbx+qc on both for 700 subjects doesn't work).

We'll have to either let go of resume, or use a strategy to lower the number of files and their sizes. This blog post describes a way to reduce file size in the work directory.

Note however that Nextflow is working on this problem right now : nextflow-io/nextflow#3849

[AlexVCaron]

This is getting more complicated on Nextflow's side :

• Automatically delete files marked as temp as soon as not needed anymore nextflow-io/nextflow#452
• Automatic task cleanup (without resumability) nextflow-io/nextflow#4713
• Overwrite published outputs only if they are stale nextflow-io/nextflow#4729

Will continue to follow