Untangling the backend QUEENS dependencies

[gilrrei]

Description of the Issue:

Over the years (to no one's fault, I was at the forefront of this 😅), we introduced dependencies between modules, which, in the current QUEENS settings, are not required and lead to strange constructs. If we unify our interpretation of the different backend modules, which we struggled to articulate as a community, these can be avoided in the future. However, while doing #237 and #243, I realised that we are closer to reaching this goal, as over the last years, thanks to the large efforts from @maxdinkel @sbrandstaeter and more, the line between modules got clearer, which drastically reduced the complexity of QUEENS and increased usability 🚀

Proposed Solution:

My personal interpretation of QUEENS modules (I would gladly discuss this with other nerds), what they should do, and what would be the difference to the current state:

(Simulation) Model

tl;dr: Multiple inputs to outputs, interface between method and simulation

• Model evaluates a function based on the input. This is quite general and acts as a bridge between iterator and simulation/modeling, the function can be Python, API related or using QUEENS' backend. It acts as the glue between front and backend
• I think models should have a parameter attribute, in order to decompose the input samples into the different parameters or fields (currently they are in the drivers for the simulation model) breaking change
• Models should decide what to do with failed simulations

Schedulers

tl;dr: Handle nested parallelism, job communication and management

• Schedulers are a resource management object. They themselves are independent who called them and who they are calling in the specific job (see Remove the need for driver objects in the schedulers #237)
• They should not be restricted to return numpy arrays (or anything) but are only an interface to allow parallelism, job management, and instruct remote connections if needed.
• They can also be used on their own without an iterator as they just handle the points above
• Schedulers should provide the job_dir (not creating the dir) to be passed to their job function. They already create the job id, so passing a valid path directly is great. This way, the callable does not need to generate their own
• Schedulers should include clean-up of its jobs, much the way clusters with SLURM or others do after running jobs. An example of this is seen in Remove empty experiment dirs #242
• Transfer data from the local to the experiment dir
• Have the ability to pass on varying data (input samples, job ids, metadata, etc) and job constant data (experiment dir, experiment name, metadata)

Drivers

tl;dr: Execute one job of the simulation tool

• Run simulations from an input data dict. breaking change
• Start 4C or other simulation codes that do not have a python API
• Create input files, output directories (relative for each job), and start the run
• Can be used without a scheduler, just runs the job
• Does not have dataprocessors 😲 (breaking change). The dataprocessor functionality is vital; however, it is not directly related to the drivers in the sense that they are not involved in the execution of the simulation tool. For a high-level API I do appreciate it, but once more, it makes for a strange interface (example gradient dataprocessor attribute)
• Handles clean up of its own files breaking change, since it knows the name of the input and output files/folders

Dataprocessors

tl;dr: Extract data from a file

• Can be used without a driver, literally should just be meant to extract data from a file, no matter what the context
• Should not have too many methods, most of them are super specific to csv
• More use of composition

Notes

Tbh, these are not big changes. Most of the time, it's shuffling functionality from one module to another. Yes, this breaks some interfaces, but IMO makes the definition clearer and avoids reverse information flows, for example, you need parameters (which is a front-end object) in the drivers (deep backend) due to the special case of random fields. This is important to allow for the split of front and backend, ultimately making the code more stable. The clear splitting up also has the advantage of reducing the need to care combinations (for example, the adjoint model does not work with a remote connection), as the different components become more independent from each other. This also helps identify difficult-to-maintain code and removes special cases from the main branch. Go wild in your scripts or projects built on QUEENS, transfer general-purpose infrastructure to QUEENS, but keep your special (and generally cool) use cases in a separate project. This also allows you to construct your own rules and objects, without having to introduce them in QUEENS.

Action Items:

No response

Related Issues:

No response

Interested Parties:

No response

[rjoussen]

Does not have dataprocessors 😲 (breaking change). The dataprocessor functionality is vital; however, it is not directly related to the drivers in the sense that they are not involved in the execution of the simulation tool. For a high-level API I do appreciate it, but once more, it makes for a strange interface (example gradient dataprocessor attribute)

Just out of curiosity: Where would you propose the data processors to be in the future? In the model? But the model doesn't even know where the driver put the results for example

[gilrrei]

Just out of curiosity: Where would you propose the data processors to be in the future? In the model? But the model doesn't even know where the driver put the results for example

I think my message was a bit cryptic 😅 What I wanted to say is that we should think of data processors as callables that retrieve results from files and implement them without considering drivers. They are useful tools that can be used by the drivers, but their inner workings or interfaces do not need to be intertwined.

[rjoussen]

Makes sense, thank you!

[shervas]

After the meeting today, we came out with these ideas:

• Discussion for model - driver - scheduler - dataprocessor refactoring #245 We agree on allowing functions in the simulation Model.
• We need more flexibility to run the driver without parameters, so the solution is splitting the function: run and run_from_dict. It adds more flexibility and should work without any further change.
• Different users use QUEENS in different ways, so we thought of creating a poll to ask QUEENS’ users what their usual workflow is. We do a general poll to be able to extract conclusions of other topics, not just drivers.
• We can follow up this poll with another poll in the QUEENS core group.
• Create a meeting to evaluate the poll results.