Layout of nested simulation driver

[rjoussen]

Description of the Issue:

It would be nice to have a driver type that, under the hood, starts a new queens experiment with an inner driver and inner iterator. Some use-cases I can imagine for that:

1. The outer iterator is a "normal" iterator, e.g., monte-carlo. Instead of a normal simulation, however, we want to perform an adaptive simulation, managed by an adaptive iterator, which starts and restarts simulations according to an adaptation rule. My primary use case for this would be to iteratively adjust the time step of a simulation mid-run in the event of simulation failure, but I believe there are definitely more types of adaptivity that would be useful.
2. One wants to wrap a more complex iterator into a grid iterator to explore settings/metadata of that complex inner iterator, e.g, do the same queens experiment for different learning rates, batch sizes,...
3. Theoretically, these two use cases could also be stacked on top of each other, i.e., the inner driver itself is an experiment driver again.

This is somewhat connected to #72 (and might close it). The reason #72 did not work is that it is not possible to start a Dask client inside another Dask client (I also think this should not be possible). To still enable nested queens runs, it is hence required that either

• The outer or inner scheduler is a Serial (or dummy) scheduler, which does not connect to a client, but instead simply runs the passed samples one by one. This requires a new Serial scheduler (which is very easy to implement, though).
• or the Dask Client created by the outer scheduler is passed down to the inner scheduler. The inner scheduler is then initialized using this already existing client. This would require changes to the initialization of the Dask Scheduler to be able to pass a Client and prevent a new Client from being created (which conveniently is already checked anyway here)

Since I am, for now, mainly interested in the use case where the inner iterator orchestrates adaptive simulations, I would stick to the first option for now, and make the inner scheduler serial.

I already have a working version of this for a simple time-adaptive (serial) 4C simulation, wrapped into a grid iterator.

Proposed Solution:

I would suggest making this a Jobscript driver to be able to utilize the file management functionality and input file management of the Jobscript base class. However, run would need to be overwritten, so some of the functionality remains unused.

Action Items:

No response

Related Issues:

#72

Interested Parties:

No response