Reimplemented Pearson's correlation to use two pass Welford's model #62750
Conversation
I've already run the relevant benchmarks. Unsurprisingly, we are looking at performance decrease in 
The problem stems from the co-moment calculations at large/small values and the asymmetric nature of Welford's Algorithm. The three values above are mathematically the same, however, when calculating the values provided in the test case, there are always two that are correct and one that is not. We could pick the value of the pair that match as a redundancy measure. Theoretically, it can only reduce our errors compared to our current version, as the values should be equal. But without a larger test pool, I wouldn't be confident enough to put it in a release version. Two-pass provides the best numerical stability at the cost of performance. |
There was a problem hiding this comment.
Thanks for running the benchmarks. I think the current implementation will lead to problems due to accumulation of roundoff errors on big datasets.
The problem stems from the co-moment calculations at large/small values and the asymmetric nature of Welford's Algorithm.
For me, the problem is in the catastrophic cancellation when computing 116.80000305175781 - 116.8000030517578.
| vx = mat[i, xi] - meanx | ||
| vy = mat[i, yi] - meany |
There was a problem hiding this comment.
One of the reasons the Welford's algorithm is considered a "stable" algorithm is by mitigating cancellation in
There was a problem hiding this comment.
The problem with the current implementation is the accumulation of round off errors. The Kahan Summation can mitigate this issue.
| - :func:`read_spss` now supports kwargs to be passed to pyreadstat (:issue:`56356`) | ||
| - :func:`read_stata` now returns ``datetime64`` resolutions better matching those natively stored in the stata format (:issue:`55642`) | ||
| - :meth:`DataFrame.agg` called with ``axis=1`` and a ``func`` which relabels the result index now raises a ``NotImplementedError`` (:issue:`58807`). | ||
| - :meth:`DataFrame.corr` now uses two pass Welford's Method to improve numerical stability with precision for very large/small values (:issue:`59652`) |
There was a problem hiding this comment.
I don't think it exists. You are simply using two-pass.
There was a problem hiding this comment.
Must have conflated the two, will update
| # Welford's method for the variance-calculation | ||
| # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance | ||
| nobs = ssqdmx = ssqdmy = covxy = meanx = meany = 0 | ||
| # Changed to Welford's two-pass for improved numeric stability |
There was a problem hiding this comment.
This comment is misleading.
|
I have thought of something that could help with performance, we can set a threshold of precision (10^-14 or 10^14) and either normalize values or apply two pass when the value is exceeded. That way we wouldn't lose performance unnecessarily. Will implement it later |
2 participants
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.This improves numerical stability for values that are really large or small such as the example given in the original issue.