np.exp() and np.log() overflow when using Softmax/Sigmoid/CrossEntropyLoss

[rrmina]

Files and classes affected:

1. activation.py

   • Sigmoid
   • Softmax

2. losses.py

   • CrossEntropyLoss

[rrmina]

So I have encountered this problem again while using eureka in training a Fully-connected layer over CIFAR-10 dataset. I found out that:

1. This problem is not caused by RMSProp.
2. This is actually a numerical stability problem.
3. This might be caused by bad initialization, which resulted to gradient explosion!
4. This is implicitly solved by using BatchNorm.

Possible Solutions:

• Penalize weight values of wider linear layers
• Traceback gradient flow in a Linear layer.