Currently runbms runs all the configurations and output . when a configuration fails. This is useful for benchmarking when we want to test all the combinations and tolerate occasional crashes, which can happen in research prototypes.
Add a flag to runbms so that the entire process exits with 1 if any invocation fails, so that CI that uses runbms doesn't need to parse the output to detect failed runs.
Currently
runbmsruns all the configurations and output.when a configuration fails. This is useful for benchmarking when we want to test all the combinations and tolerate occasional crashes, which can happen in research prototypes.Add a flag to
runbmsso that the entire process exits with1if any invocation fails, so that CI that usesrunbmsdoesn't need to parse the output to detect failed runs.