xray/Shotset: out of memory?

[CoChrists]

Citing from the api docs:
"The key power/functionality of Shotset is that it provides a layer of
abstraction over the intensity data on disk. Specifically, it provides many
powerful capabilities for analyzing data sets that are larger than can
fit in memory."
However, when I want to create 20000 shots via simulate() then I get:

[...]
File "/usr/local/lib/python2.7/dist-packages/thor/xray.py", line 1728, in simulate
ValueError: array is too big.

With 5000 shots it works without complains...

So the promise does not stand?

[tjlane]

Ah good point, this is a feature that should probably be added. Once the shotset or rings object is created, it should work fine operating on disk, but you can see that when simulate is called it attempts to create an array in memory:

https://github.com/tjlane/thor/blob/master/src/python/xray.py#L1728

This isn't too hard of a fix, though. If you want to take a shot at it, feel free to create a new branch and open a pull request. If you don't feel up to it, I'll fix it (eventually). Here's an example of the kind of code you want, using pytables:

https://github.com/tjlane/thor/blob/master/src/python/xray.py#L3528

As a short term solution, just break your simulation into smaller chunks and recombine them later. I noticed that I still need to implement the Shotset add function, so I can open a separate issue to take care of that sooner rather than later.

Final note: the Rings class lets you do simulations directly in Fourier space, which is often much more efficient. The only reason to use Shotset for CXS simulations is really if you want to see what the image would look like when you actually do an experiment. Rings has an append method

https://github.com/tjlane/thor/blob/master/src/python/xray.py#L3600

That's useful for splitting up simulations across many machines & then recombining them.

[CoChrists]

Simulation of 20000 shots using the polar detector works. Thanks for mentioning the Rings class!

[CoChrists]

Unfortunately, after running the Rings.simulate() with 20000 shots I get this:

[...]
Finished polar shot 19999/20000 on device 0
Finished polar shot 20000/20000 on device 0
Traceback (most recent call last):
  [...]
  File "/usr/local/lib/python2.7/dist-packages/thor/xray.py", line 3499, in simulate
  File "/usr/local/lib/python2.7/dist-packages/thor/xray.py", line 2371, in __init__
  File "/usr/local/lib/python2.7/dist-packages/numpy/lib/function_base.py", line 830, in copy
    return array(a, order=order, copy=True)
MemoryError