Each stage should check that it completed

[necrolyte2]

If each pipeline can simply check itself such that if any of its subcommands fail then the entire stage can fail and then the pipeline itself can error and know exactly where it failed.

• step1
• host_map
• quality_filter
• ray2_assembly
• iterative_blast_phylo

[averagehat]

It looks like Ruffus is capable of doing this http://www.ruffus.org.uk/tutorials/new_tutorial/checkpointing.html

We are already using Ruffus but luigi is another framework that does this. Luigi (and probably ruffus?) also makes it easy to run stages individually (#265)

[necrolyte2]

That is correct, but with this pipeline it was a bit harder to just do this since all the scripts were dependent on a specific directory structure and were not installing into the virtualenv itself which made portability hard.

Now that #265 is nearly complete replacing pathogen.pl and run_standard.pl in the future with ruffus tasks will be possible where it was not prior.

[averagehat]

As discussed one option would be to use a Makefile template and populate it with options present in a config.yaml file. Similar to the idea of ngsxml and ./configure and automake.