Walltime of cluster scheduler

[bwirthl]

I have a short question: The walltime I set in the cluster scheduler, is this the walltime of the worker or the simulation? The documentation says Walltime for each worker job. How do I set it properly? Let's say my simulation takes less than 20 min, I have 100 samples and 5 in parallel. Thanks for your help!

[maxdinkel]

The walltime (+ some extra minutes) is basically the lifetime of one worker. When the lifetime is over, the worker is restarted. Assuming that you set restart_workers=False, I would propose to set the walltime to around 12h to 24h (100 / 5 * 20min * security_factor). If, for some reason, the walltime is not sufficient for all jobs, the unfinished jobs are restarted on the restarted workers.

[sbrandstaeter]

The wall time should be the wall time of the Slurm job that runs the Dask worker.

The recommendation on how to choose it critically depends on whether you want to use the restart_worker feature or not.
The case in which you set restart_worker=True is easier; the wall time for the worker should be set a little above the expected runtime of a single simulation in your example, 20 min.
In case you use restart_worker=False, it becomes a bit more difficult to estimate the recommended wall time. In your example, I would say the following: assuming that you really get your five Dask workers for the entire time, the ideal wall time is 100/5 * 20 min = 400 min. Problems begin when you dont get all the resources you were planning with, i.e., less than 5 workers.
On the other hand, when your worker runs out of time, it should be closed automatically, and the Dask scheduler should try to start a new worker. I have not tested this, though.

[bwirthl]

@maxdinkel and @sbrandstaeter Thank you very much for your answers! restart_worker=True is the answer I was looking for.