Add benchmarks with regression testing #815
Conversation
|
An example plot from 
|
mcgibbon
changed the title
AnnaKwa
left a comment
AnnaKwa
left a comment
There was a problem hiding this comment.
Cool feature! I (ok, mostly cursor) was able to use this to generate a benchmark for the diffusion UNet in an experiment branch. I had a few minor comments.
| ) | ||
|
|
||
| @classmethod | ||
| def _new_with_params( |
There was a problem hiding this comment.
What's the purpose of having this separate method rather than just having this in new?
There was a problem hiding this comment.
So that it can be re-used by the two different new implementations.
|
|
||
| from fme.core.benchmark.benchmark import get_benchmarks | ||
|
|
||
| RESULTS_PATH = pathlib.Path(os.path.abspath(os.path.dirname(__file__))) / "results" |
There was a problem hiding this comment.
It'd be nice to print at the end of main where your png is saved.
|
|
||
| from fme.core.benchmark.benchmark import get_benchmarks | ||
|
|
||
| RESULTS_PATH = pathlib.Path(os.path.abspath(os.path.dirname(__file__))) / "results" |
There was a problem hiding this comment.
Could this be an optional CLI arg? The use case I'm thinking of is if I wanted to run and save the results to /results before a job starts on a cluster with different hardware since it's otherwise a pain to start an interactive job just to get that information.
There was a problem hiding this comment.
Added as a CLI arg.
|
Added a basic regression test just to make sure it still runs. |
2 participants
This PR adds a system for registering benchmarks to be run in a new
fme.core.benchmark.runentrypoint.All benchmarks are regression tested for backwards compatibility.
Changes:
Added BenchmarkABC and register_benchmark to record new benchmarks
Added
fme.core.benchmark.runentrypointconftest.py updated to use deterministic algorithms during tests
Added
fme.core.device.force_cpucontext to force cpu usage during regression testsTests added