Support to complex numbers

[GiggleLiu]

Optimization schemes with complex numbers are widely used in physics, and recently, machine learning.

I strongly suggest to add the support to complex numbers for optimization engines like RmsProp et. al.

We just need a few lines of change and several tests.

E.g. climin/rmsprop.py line 165-167

            self.moving_mean_squared = (
                self.decay * self.moving_mean_squared
                + (1 - self.decay) * gradient ** 2) 
            --> + (1 - self.decay) * np.abs(gradient) ** 2)

A single line of change would make it applicable for complex numbers.

The same is true for Adam and Adadelta.

On the other side, GradientDescent works well already without any change.

Maybe a bit effort is needed for Rprop, I have no clue yet how to make it compatible with complex numbers due to the ill defined sign function for complex numbers.

[GiggleLiu]

A possible solusion to Rprop that works well in my own test:

file: climin/rprop.py, line 129-135

            gradprod = gradient_m1 * self.gradient   
        --> gradprod = gradient_m1.conj() * self.gradient 

            self.changes[gradprod > 0] *= self.step_grow
            self.changes[gradprod < 0] *= self.step_shrink
            self.changes = ma.clip(self.changes, self.min_step, self.max_step)

            step = -self.changes * ma.sign(self.gradient) 
        --> to step = -self.changes * np.exp(np.angle(self.gradient))

About Pull Request

I feel sad to noticed that gnumpy do not support complex numbers yet, so I will not create a pr now.