OSError: Failed to open shared memory

[mekman]

Hi,

thanks for sharing this package! On a fresh installation (tested both Mac OS X and Linux) I'm getting this error.

In [17]: np.__version__
Out[17]: '1.9.0'

In [18]: import sharedmem as shm

In [19]: array = shm.zeros((300,300,1), dtype=float) # works fine

However, this results in the error posted below. Do you have any idea what might be causing this?

In [20]: array = shm.zeros((300,300,10), dtype=float)


---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-20-cbadca5ba769> in <module>()
----> 1 array = shm.zeros((300,300,10), dtype=float)

/Users/mekman/anaconda/lib/python2.7/site-packages/sharedmem/array.pyc in shared_zeros(shape, dtype, order)
    132     """ Create a shared-memory ndarray filled with 0s.
    133     """
--> 134     arr = shared_empty(shape, dtype, order)
    135     x = np.zeros((), arr.dtype)
    136     arr[...] = x

/Users/mekman/anaconda/lib/python2.7/site-packages/sharedmem/array.pyc in shared_empty(shape, dtype, order)
    123     size = int(np.prod(shape))
    124     nbytes = size * dtype.itemsize
--> 125     wrapper = heap.BufferWrapper(nbytes)
    126     strides = None
    127     offset = 0

/Users/mekman/anaconda/lib/python2.7/site-packages/sharedmem/heap.pyc in __init__(self, size)
     63     def __init__(self, size):
     64         self.size = size
---> 65         self.offset, self.buf = _heap.allocate(size)
     66 
     67     def __setstate__(self, _state):

/Users/mekman/anaconda/lib/python2.7/site-packages/sharedmem/heap.pyc in allocate(self, size)
     50         with self.lock:
     51             if size > mmap.PAGESIZE:
---> 52                 offset, buf = self._allocate_big(size)
     53             else:
     54                 offset, buf = self._allocate_tiny(size)

/Users/mekman/anaconda/lib/python2.7/site-packages/sharedmem/heap.pyc in _allocate_big(self, size)
     42 
     43     def _allocate_big(self, size):
---> 44         buf = SharedMemoryBuffer(size)
     45         addr, _ = buf.getbuffer()
     46         return 0, buf

/Users/mekman/anaconda/lib/python2.7/site-packages/sharedmem/sharedmemory_sysv.so in sharedmem.sharedmemory_sysv.SharedMemoryBuffer.__init__ (sharedmem/sharedmemory_sysv.c:2759)()

/Users/mekman/anaconda/lib/python2.7/site-packages/sharedmem/sharedmemory_sysv.so in sharedmem.sharedmemory_sysv.SharedMemoryBuffer.__init__ (sharedmem/sharedmemory_sysv.c:2491)()

OSError: Failed to open shared memory

[sturlamolden]

At least I can reproduce the error.

I dont't really know what could be the problem, because I have mostly worked on the Windows version. I don't have time to look at it right now, but if you find it please send me a PR.

Today I would have used posix shm_open and mmap instead of System V IPC though. So in any case it is time for a major rewrite.

On Linux you also get shared memory by memory mapping from /tmp, but it does not give you shared memory on Mac.

[sturlamolden]

It fails with errno 12 (allocation failure)

[sturlamolden]

I can also produce errno 35 (EAGAIN), which is an error that shmget should never set.

[sturlamolden]

Maybe it is time to just take this old code down, because it does more harm than good.

[mekman]

Thanks for taking a look! Too bad, sharedmem sounded exactly like what I needed :)

Do you happen to have any experience with joblib and might know whether similar things wrt to shared memory can be achieved with that package?https://github.com/joblib/joblib/blob/master/doc/parallel_numpy.rst

Or maybe you have other pointers?

[sturlamolden]

Joblib memory maps from /tmp to allocate shared memory. It means it don't use shared memory on Mac, only Linux.

[mekman]

Great, thanks for your comments.