Flaky Tests Question

[jordan-gillard]

Flaky tests, such as test_conditional_prob_inf_given_vl_dist, are due to their non-deterministic nature. Monte Carlo simulations, such as those used in the baseline_exposure_model fixture and test_conditional_prob_inf_given_vl_dist, fail because they're probabilistic. For modelling this is great - but for unit tests its not 😅 It's a nice feeling to have a 🟢 CI.

All you'd have to do to fix this test is set the random seed, like np.random.seed(42). However, this is misleading if the goal (as I suspect) is to ensure the models accuracy within a certain tolerance.

If the goal is to measure the models accuracy, what if you removed the 3x retry and instead ran the model 10 times, gathering the results each time? Then you could do a statistical analysis against the mean/median/standard deviation/percentiles etc.

Another alternative is to change the fixed 0.002 tolerance. What if you calculated the tolerance based on the number of runs? Could an absolute tolerance of 0.002 be wishful thinking?

It could also help to log model deviations and store them with a timestamp. Then you can monitor deviations over time.

I'm happy to fix this test (and get that CI ✅). Its a cool project. Just let me know what the expected behavior is & how I can help out.

[jordan-gillard]

😱 y'all are on GitLab? Which repo should I create PRs for?

[sanchalitorpe-source]

Thanks for opening this — I agree that the flakiness here is coming from the Monte Carlo nature of the model rather than a true bug.

I see a few possible ways to address this and wanted to check which aligns best with the intended purpose of the test:

1. Make the test statistically robust by running the model multiple times and asserting on summary statistics (e.g. mean / percentile bounds) instead of a single outcome. This would preserve the probabilistic nature while avoiding flakiness.

2. Fix the random seed within the test to make CI deterministic, with a clear comment that the test is checking regression rather than statistical accuracy.

3. Replace the fixed absolute tolerance with one derived from the number of simulations / expected variance.

I’m happy to implement whichever approach is preferred — just wanted to confirm the expected behavior of this test first.