bug; the CAE penalty is incorrect for any non-sigmoid activation

[Sycor4x]

The CAE class uses the relu nonlinearity. https://github.com/avijit9/Contractive_Autoencoder_in_Pytorch/blob/master/CAE_pytorch.py#L61

However, the way the CAE loss is computed is only valid for sigmoid activations. This is easy enough to show: the derivative of the relu activation is 1 for positive inputs and 0 for negative inputs.

The link provided in the docstring for the CAE loss assumes the sigmoid nonlinearity; it's not attempting to derive the contractive penalty in general.

[haQiu-oMNi]

I agree with @Sycor4x . The contractive_loss formula used in this code is only valid for logistic sigmoid activation function. However in this code the activation function of the hidden units is ReLU.

[DaikiTanak]

I agree with @Sycor4x and @haQiu-oMNi too. Approximation in this code doesn't hold for non-linear encoder which has ReLU as an activation function.

[Sharp-rookie]

I agree with @Sycor4x, @haQiu-oMNi and @DaikiTanak . Could anyone provide the collection of CAE loss of all types of activations?

[Sycor4x]

@Sharp-rookie The way to write the code to compute the CAE loss for arbitrary activations is to use PyTorch to compute the Jacobian for you, and then compute the norm from there. It's not hard to code up. Unfortunately, I don't have access to the code that I wrote to do it anymore.