Slurm support

[aaronk]

At the request of a colleague here, I've done a rearrangement of the cluster code to support Slurm.

I did a fairly substantial refactoring of the cluster code to make it easier to customize the support for a given system. Basically I took the cluster_utils functions (e.g. launch_job) and set them up on some factory managed "cluster engine" classes. The factory chooses the engine based on the cluster command entry in the settings file (e.g. "bsub" returns the LsfClusterEngine).

One of the reasons that I did this is that running Slurm jobs in our environment really didn't fit in well with the existing code. Given Slurm's requirement that memory be specified; our internal requirement for a time setting; the different possible partitions and module loads that we might need; I chose to set a jobscript template via miso_settings into which the command is substituted. This should provide maximum flexibility for submitting jobs without requiring all sbatch parameters be supported individually by the settings file.

[yarden]

There's a simpler way to do this. All you'd need is a wrapper script like send_job that takes a shell script as input and submits it to the cluster, passing it whatever parameters are needed. Then you just register send_job as the cluster submission script in the MISO settings file.

It doesn't make sense for us to support every cluster system and its quirks, especially since if I incorporate this code, I'll have no way of testing it (I have no access to Slurm clusters.)