Metadata falsely states that jobscript was successful

[leahaeusel]

Description of the Issue:

If you use the Jobscript driver with raise_error_on_jobscript_failure=False and the jobscript does indeed fail during the QUEENS run, the metadata falsely states that run_jobscript was successful.

Proposed Solution:

Instead, the metadata should indicate that run_jobscript failed, even though no error was raised.

Action Items:

No response

Related Issues:

No response

Interested Parties:

No response

[gilrrei]

Some problems regarding handling errors:

• A failing simulation does not mean the entire run should fail
• There are different failure reasons, from convergence problems to incorrect input files
• It's difficult to track the errors from simulation software since:
  • Some write to stdout, some to stderr, some to both
  • Some codes raise exit codes, even though nothing failed
  • Sometimes the MPI setup isn't perfect, such that it raises errors even in a successful call.
• The points above are the reason we have raise_error_on_jobscript_failure

These are problems that have been around for some time, and the only answer I could find is that it is a case-specific problem.

The metadata only logs the error-free execution of a Python code, which it does if no error is raised in the subprocess; the function call is marked as successful. I completely agree that this is super misleading, and we should change this. A simple workaround if you have a dataprocessor is to grep for the output; if it's Null, Null, it means nothing could have been read in.