change to pmap step in computeNextIteration

[gregobad]

Hi -

I modified the pmap step in computeNextIteration! to only parallellize the objective function evaluation step (not the accept / reject or proposal steps). This avoids having to pass the entire chain to the workers. I was running into problems where the evaluation slowed down over time and eventually failed with a bus error because the chain objects get very large. I think this should solve that problem.

Greg

[floswald]

sounds like a good idea! thanks!

[gregobad]

This actually doesn't go far enough. pmap still gets passed a reference to the chain object, which is growing over time. Will update shortly to fix that.

[floswald]

hey how are we doing with this? good to merge?

[gregobad]

This change by itself doesn't give much of a memory improvement. To get any improvement it's necessary to use CachingPool. I couldn't figure out a way to do this cleanly. My current version of the code defines a global pool wp which gets passed to pmap in computeNextIteration!:

evs = pmap(x -> evaluateObjective(algo.m, x), wp, pps)

But this solution doesn't make a lot of sense for library use.

[gregobad]

I've added an option to AlgoBGP.opts, "worker_pool", that allows the user to specify a worker pool to use for distributed computation. This gets passed to the pmap step in computeNextIteration!. This branch also has a few other additions, like the possibility of passing additional arguments to the objective function as keyword options.

[floswald]

hey @gregobad , thanks for this. just checking, did you run the tests locally?

[floswald]

🎉

[gregobad]

I didn't run the tests locally but looks like they worked. I thought I was just pushing to my fork, didn't intend to create a pull request on the main branch. But perhaps the vector setMoments! is generally useful.

[floswald]

sure we'll keep the vector version around, why not. took some tweaking to make the test run on x86.