Add input argument to control custom python envs

[neergaard]

First off, this is great and much better than what I had made for myself.

Second, something that I thought was nice was the option to specify a conda environment to activate before running the notebook. I've implemented something basic, but I don't know it is something that is good for general use.

[raphtown]

Hey aneergaard! If you have an sbatch script that uses conda environments + notebooks, you can totally submit it as a PR. Probably just have the conda env specified as an additional input to the sbatch script.

[vsoch]

hey @aneergaard ! I've taken over maintainer for this repo, and I think the repo2docker-conda would fit the bill for you to use, and then use the sherlock/singularity-notebook.sbatch with it. Do you want to give it a try? Here is the repo2docker-conda template you would want to start with --> https://github.com/vsoch/repo2docker-conda

[zqfang]

thank you very much for this repo @vsoch. I've manngered to use my own conda env and train my model. Follow these steps, you could use any conda env you want.

• Step1: install Miniconda on sherlock, and create the conda env you want
• Step2: add different envs to the jupyter kernel
• Step3: create your jupyter.gpu.sbatch like this

#!/bin/bash

PORT=$1
NOTEBOOK_DIR=$2
if [ -z "$NOTEBOOK_DIR" ]; then
    cd $SCRATCH
else
    cd $NOTEBOOK_DIR
fi

module load gcc/7.3.0
module load gdb
module load cmake
module load cuda/10.1.168
module load cudnn/7.6.4
module load nccl

conda activate fastai # use the env you've created !
jupyter lab --no-browser --port=$PORT # I use jupyterlab

[vsoch]

This is really useful! Do you want to add a recipe, or write this up as a note in the README documentation?

[zqfang]

I just create my own conda envs and follow your tutorial. I could make a PR if you need a recipe @vsoch

[vsoch]

I'd love that, if you have the bandwidth. I think your derivation (with cuda) is more akin to what would be a real world example, and helpful for others! There is definitely no rush, but I'll be excited to share widely when it's ready. I can also offer to help however you might need it, hopefully should be fairly straight forward to write the markdown!

[zqfang]

@vsoch , I've add a recipe in my repo: https://github.com/zqfang/forward/blob/master/recipe.sherlock-gpu.md .

Where should I put this file? I could submit a PR if needed

[vsoch]

I think it would be appropriate to add to the Tiny Tutorials section - what do you think?

If you don't intend to keep the repository, you can add an examples folder at the root here and drop the file there (and link from tiny tutorials) otherwise, a link should work :)