Clean up _manage_paths in the driver object

[sbrandstaeter]

Description of the Issue:

The current implementation of the managed paths method in the driver object is overly specific and includes functionality that’s mainly relevant for 4C-related models.

Refactoring the _manage_paths logic to be more general would help make the driver object cleaner and easier to maintain. After moving the 4C-specific logic, additional adjustments may be needed.

Proposed Solution:

For example, the handling of the output file is tightly coupled to 4C-specific behavior, which should instead be moved to the 4C interface module.

Action Items:

No response

Related Issues:

#170

Interested Parties:

No response

[gilrrei]

I just found an old branch, where I changed this. Not only is the manage path stuff related to 4C, but the driver is also strongly linked to injection. I think the way to go would be to make the prepare_input_file a user input, with the signature

    def prepare_input_files(
        self, sample_dict: dict, job_data: dict, experiment_dir: Path, job_dir: Path
    ) -> tuple[dict, Path, Path]:

Which creates the input files and returns:

• job_data a dict with all the relevant inputs to inject in the jobscript
• output_dir a path to where the output is (needed for the dataprocessor)
• log_file a path to tell the driver where to put the log file

An example of this is injection see here:

class Injection(JobPreprocessor):

    def __init__(self, input_templates: dict, files_to_copy: list = None):
        if not isinstance(input_templates, dict):
            input_templates = {"input_file": input_templates}

        self.input_templates = {
            input_template_key: Path(input_template_path)
            for input_template_key, input_template_path in input_templates.items()
        }
        if files_to_copy is None:
            files_to_copy = []

        files_to_copy.extend((str(file) for file in self.input_templates.values()))

    def prepare_input_files(self, sample_dict, job_data, experiment_dir, job_dir):

        output_prefix = "output"
        output_dir = job_dir / output_prefix

        output_dir = create_folder_if_not_existent(output_dir)

        output_file = output_dir / output_prefix
        log_file = output_dir / (output_prefix + ".log")

        # Additional data
        job_data = job_data | {
            "output_dir": output_dir,
            "output_file": output_file,
            "log_file": log_file,
        }

        # Create the paths for all input files
        for input_template_name, input_template_path in self.input_templates.items():
            input_file_str = input_template_name + "".join(input_template_path.suffixes)
            job_data[input_template_name] = job_dir / input_file_str

        # Inject all the data into the all the input files
        for input_template_name, input_template_path in self.input_templates.items():
            inject(
                job_data | sample_dict,
                experiment_dir / input_template_path.name,
                job_data[input_template_name],
            )

        return job_data, output_dir, log_file