Convergence tests should fail when convergence rates get too high

[mwarusz]

I am looking into #472 and got the following result from ocean/spherical/qu/rotation_2d on pm-cpu with gnu:

Order of convergence for tracer1: 75.975
Order of convergence for tracer2: 60.361
Order of convergence for tracer3: 61.226
          execution:        ^[[92mSUCCESS^[[0m
          runtime:          ^[[94m0:00:03^[[0m
POLARIS TASK: PASS

These convergence rates are ridiculous and the test should fail.

[xylar]

@mwarusz, we currently haven't bothered with an upper bound as we haven't encountered anything like this before. It's hard enough to maintain reasonable lower bounds. I don't know that we want to curate a corresponding set of upper bounds. if you think it would be useful to have a ludicrous upper bound, we could add that but again, it's just not a common problem.

[mwarusz]

When I did convergence testing in other projects, we were usually checking that a convergence rate is between some target rate +/- tolerance (e.g. 2.0 +/ 0.2). If we can find a common tolerance that works for most of the convergence tests in Polaris, then it may not be that much work to maintain. However, I can see how it can be difficult if the tolerance needs to be adjusted for every convergence test.

[cbegeman]

It seems reasonable to add max_allowable_convergence_rate config option for all tests. However, I'd put this as a lower priority given that I hope people would get test case fails at least due to a significant difference from baseline.