Multiple mission runs and reports directory

[hschilling]

Description

When running aviary/examples/run_off_design_example.py there are some issues with the reports folders.

1. Since there are 3 missions runs in the script, there should be 3 reports folders but there are only 2. The third mission run overwrites the directory of the second mission run
2. After the second mission run, the directory has a full set of reports. But after the third run, it has a greatly reduced set of reports.

The normal set of reports is

input_checks.md
inputs.html
mission_summary.md
mission_timeseries_data.csv
n2.html
opt_report.html
overridden_variables.md
status.json
subsystems
total_coloring.html
traj_linkage_report.html
traj_results_report.html

but after the third run, only get

driver_scaling_report.html
n2.html
total_coloring.html
traj_results_report.html

Example

Run

python aviary/examples/run_off_design_example.py

Aviary Version

0.10.0.dev0

Relevant environment information

No response

[OwenShepherd]

Hi hschilling. If you can, it would be helpful to provide the commit hash for the specific version you're working with while the project is under active development. I.e. 0.10.0.dev0 could apply to many different versions of the software if you're pulling off of the main branch.

[OwenShepherd]

@jkirk5 on my pull request #981 you mentioned that you would like a more robust solution to this issue. I've opened up a draft pull request #983 that uses the following method for naming:

For off design problems without a name the problem type and the time the problem was run will be added to its name to produce the following directory outputs:

• test_fallout_020526182200_out
• test_alternate_020526182223_out
• and so on.

Is this more aligned with what you were looking for?

[jkirk5]

Is this more aligned with what you were looking for?

Hey @OwenShepherd, yeah this was more in line with what I was planning to do. Sorry I haven't gotten around to reviewing that PR yet, I will do that now! Thanks for your contributions and continued engagement w/ the repo

[jkirk5]

Ok TLDR; your PR is fine as-is, but I think we can make it more readable for users, and if the team comes to a decision on how we can do that now, otherwise we can just update that later.

I am concerned that the folders become this hard-to-parse string of numbers that will seem like random/cryptic code to new or basic users. This implementation needs to record down to the second in case someone runs a lot of quick cases back-to-back, but other than that I don't think users care about that level of detail.

I am leaning towards copying OpenMDAO's pattern of simply incrementing the name by +1 each repeat, and we can capture runtime and other important information elsewhere, maybe some kind of new report with meta info about the Aviary run. Such as run start & end time, who ran it, what input files were provided, and how Aviary was accessed. Sorry for the ramble, just getting my thoughts recorded here!

[OwenShepherd]

I am leaning towards copying OpenMDAO's pattern of simply incrementing the name by +1 each repeat, and we can capture runtime and other important information elsewhere

I had a solution that behaved this way, but didn't like checking the filesystem as part of the sequential naming (i.e. only increment if that report directory already exists). I chose the timing for something more predictable and figured verbose naming wasn't a huge concern, but thinking some more about it, naming based off of the timing would cause issues if multiple off-design runs are completed within the same second, and thus does not actually fix this bug / issue. So, probably better to use sequential naming.

I'm not entirely familiar with how users are expected to perform off design missions. I.e., would it be expected for me to set up an Aviary problem, run it, then run some off-design missions using that same basic Aviary problem within the same script? If so, if I created another solution I would probably just add a counter to aviary.interface.methods_for_level2.AviaryProblem and name the design runs based off of that counter. But that's just me trying to avoid filesystem checks from affecting the naming.