Overflow that breaks learning #2
Description
During training, the following happened:
/home/andrew/git/numpy-saliency/model/layers.py:234: RuntimeWarning: overflow encountered in exp
exp = np.exp(inputs, dtype=np.float)
/home/andrew/git/numpy-saliency/model/layers.py:235: RuntimeWarning: invalid value encountered in true_divide
self.out = exp/np.sum(exp)
/home/andrew/git/numpy-saliency/model/layers.py:214: RuntimeWarning: invalid value encountered in less
x[x < 0] = 0
/home/andrew/git/numpy-saliency/model/layers.py:219: RuntimeWarning: invalid value encountered in less
dx[self.inputs < 0] = 0
After this warning, training continues, but model performance degrades entirely. Before the error, this network was getting Batch Accuracy ~ 96%
