Potential Future Solvers/Algorithms

[rcarson3]

I'm just going to keep a list here of solvers/algorithms that might be interesting to implement in SNLS. I'll keep adding to this list as I find more that might be of interest.

I cam across this one today: https://arxiv.org/abs/2112.02089 which could be seen as a variation of a Levenberg-Marquardt type solvers. I was just looking through it (alg 2 mainly), and it seems like it would fairly simple to implement given it's not too far off from what we already do. If the convergence properties hold for our type of problems this could be a very interesting one to use especially for our very stiff equations.

Another one that might be of interest would be potentially making use of something like the double-dogleg in our solvers. You can find a description of it at section 6.4 of https://doi.org/10.1137/1.9781611971200.ch6 or https://doi.org/10.1007/BF00932218 . Although, it seems that this is largely the same as the single dogleg with the caveat that earlier on it would be a bit more biased towards the newton step rather than the cauchy point. However, I'm not necessarily sure it would be worth adding unless it drastically seems to help our solvers, which I don't believe it will.

[nrbarton]

Another option that you could consider is the subspace version of the trust region subproblem. In the subspace variant, you solve a "full" trust region problem (meaning an eigenvalue solve) but in a subspace. A common enough approach is to have the subspace span the Newton step and the steepest descent direction -- so the same subspace as you traverse in the usual dogleg approach. Subspace might have a mild advantage over dogleg in for some problems ... but it has been too long since I have looked at it for me to remember the details.

[rcarson3]

So, I think one other avenue is looking at simpler ways methods to control the delta as I've found that the new hybrid solver and the original trust region dogleg solver behave very similarly for certain problems. While we do want them to roughly the same solution, I have found for some problems using a more heuristic way of doing the delta control can be beneficial. However, I guess that's just the fun-ness of dealing with highly nonlinear problems :/