Efficiency

[eric-tramel]

It would be nice if some of you speed demons (@amanoel, @alaa-saade) could take a look at the code in rbm.jl and make sure that we're computing the RBM training in an efficient way!

[dfdx]

There are basically 3 ways to go here. First, the most simple and the least efficient is to optimize specific blocks of code. I would start with logistic function, since it's one of the bottlenecks. Maybe Devectorize package can give some improvements, but I doubt you will get more than 30% speedup.

Much better way is to integrate the package with GPU. I had some bad luck with it previously, every time ending up with re-installing OS, but potentially it may make it 8-10 times faster.

Last option is to bring RBM to a cluster. I have a long-term idea of implementing distributed version of RBM, but it's quite a complex goal, so wouldn't expect it soon.

[dfdx]

Also note, that it's possible to optimize fitting algorithmically. E.g. I found that persistent CD together with appropriate momentum can speed up training quite a lot - individual epochs will take the same time, but you will need much smaller number of them.

[krzakala]

Well do you use the Appe accelerate to compute the exp for the sigmoid?

F

—
Sent from Mailbox

On Wed, Sep 9, 2015 at 6:22 PM, Eric W. Tramel notifications@github.com
wrote:

It would be nice if some of you speed demons (@amanoel, @alaa-saade) could take a look at the code in rbm.jl and make sure that we're computing the RBM training in an efficient way!

Reply to this email directly or view it on GitHub:
#3

[eric-tramel]

@dfdx Thanks for checking in on our fork! We really liked that you had taken the initiative to pull together an RBM package on Julia. We thought we'd pull together some more features and add in some research we've doing :)

Much better way is to integrate the package with GPU. I had some bad luck with it previously, every time ending up with re-installing OS, but potentially it may make it 8-10 times faster.

We agree with this approach, though, as you say, it can get messy quickly! There was a lot of work done for the torch7 implementation by @skaae to implement an RBM in this way, so perhaps he would have some insight into the general structure that could then be utilized in similar fashion within Julia.

Last option is to bring RBM to a cluster. I have a long-term idea of implementing distributed version of RBM, but it's quite a complex goal, so wouldn't expect it soon.

It would, again, take some thinking about the proper points of parallelization. In this case, one could distribute the per-mini-batch computations to different nodes in the cluster. The parallelization would be on the order of the number of samples in a batch, since one still needs to aggregate the samples together to derive the approximate gradient vector for the steepest update on the RBM model parameters before moving on to the next batch.

Also note, that it's possible to optimize fitting algorithmically. E.g. I found that persistent CD together with appropriate momentum can speed up training quite a lot - individual epochs will take the same time, but you will need much smaller number of them.

It is quite true that one can toy with the tuning parameters to reduce the training epochs. However, this is quite dependent upon the training set. Other changes can be made, algorithmically, to reduce the number of epochs required for training a generalized generative RBM model, but this is ongoing and active research in the field :)

[eric-tramel]

@krzakala:

Well do you use the Appe accelerate to compute the exp for the sigmoid?

From the reports from @amanoel, we've seen that one can get a boost by using the AppleAccelerate.jl for exp calculations, so I'm sure it would give an advantage, here, too.

The problem is that this is a binding to an OS X-specific toolset, and so, a naive implementation would restrict this fork of Boltzmann.jl to only work on Macs. Perhaps, though, since so much of the user-base (in our lab at least) is working on a Mac, we could include a switch to check for the user's system, and if its a Mac, then we could call the AppleAccelerate implementation and use the default one otherwise.

This way we could squeeze a bit of power for some machines without breaking anything.

[krzakala]

@eric sure, you are right, this was what I had in mind!

[eric-tramel]

@krzakala:

I took a first stab at using AppleAccelerate.jl for the logistic sigmoid operation. This work currently lives on the accelerate branch. I don't have everything linked into the main training script yet, when I do I'll merge it together.

Right now, the AppleAccelerate version of the logsig operation looks lie

function logisticAccel(x)
    return AppleAccelerate.rec(1+AppleAccelerate.exp(-x))
end

Here, the function AppleAccelerate.rec() is simply the reciprocal operation. I made a test script (test/testaccel.jl) from which I was able to get the following timing comparison between the base implementation and the accelerated version...

Testing AppleAccelerate Logsigmoid (25 Runs)
===================================================
   [NoAccel] Vector (65536 x 1): 0.049372756 sec
   [NoAccel] Matirx (256 x 256): 0.047258585 sec
   [Accel]   Vector (65536 x 1): 0.023749814 sec
   [Accel]   Matirx (256 x 256): 0.021677888 sec
===================================================
Vector Accel Calculation: 2.078869165038513 Times Faster
Matrix Accel Calculation: 2.180036403915363 Times Faster
===================================================

So it seems like Mac users get a free 2x improvement in calculation time. After I verify that I'm not doing anything wrong in terms of accuracy, I'll put the necessary flags into the main code and merge this freebie into the main branch.

[eric-tramel]

In the documentation for AppleAccelerate, there doesn't seem to be any mention if

using AppleAccelerate

on a non-OSX machine throws and error and halts execution or not. Ideally, it should throw some warning gracefully. I'd love it if this were the case. Better yet, if there is no warning except for when the functions are called. Since we'll have checks in place to make sure only OS X systems get that far, I would hope PC/Linux users would never notice the difference.

@dfdx: Are you on a PC? Would it be possible for you to verify the operation of AppleAccelerate ?