How to deal with numerical instabilities

[coding-ecoli]

Hi Dr. Will Townes,

When I use NSF, I often face numerical instabilities during training. In the training loop, you implemented the maximum number of tries; however, the model sometimes exceeds the number of tries and needs to restart. I could set the maximum number of tries higher, but I am more interested in why these numerical instabilities happen.

I would be interested to know if you have encountered this numerical instability issue before and how you've fixed it or if you have any recommendations on what to do when facing numerical instabilities. Are there some parameters that I can experiment with or need to be extra careful about?

Thanks and best,
Wangjun