On Windows, gputools and libcudaDeconv compete for GPU: OSError: [WinError -529697949]

[tlambert03]

tracking a bug seen on some windows computers (definitely seen on win10, maybe win7) in which importing gputools causes a crash when trying to access shared functions in libcudaDeconv.

>>> import llspy
>>> import gputools  # comment out this line and it succeeds
>>> previewPath = 'basic_sample'
>>> otfdir = 'OTFs'
>>> E = llspy.LLSdir(previewPath, True)
>>> llspy.llsdir.preview(E, 0, 0, otfDir=otfdir)
"""
WARNING:gputools.config.myconfigparser | [Errno 2] No such file or directory: '~/.gputools'
output nz=84
cufftPlan3d() c2r failed

OSError                                   Traceback (most recent call last)
<ipython-input-1-c354b9e601b9> in <module>()
      5
      6 E = llspy.LLSdir(previewPath, True)
----> 7 llspy.llsdir.preview(E, 0, 0, otfDir=otfdir)

~\Dropbox (HMS)\Python\llspy\llspy\llsdir.py in preview(exp, tR, cR, **kwargs)
    197             for i, d in enumerate(zip(stacks, P.otfs)):
    198                 stk, otf = d
--> 199                 stacks[i] = quickDecon(stk, otf, **opts)
    200         else:
    201             # deconvolution does deskewing and cropping, so we do it here if we're

~\Dropbox (HMS)\Python\llspy\llspy\libcudawrapper.py in quickDecon(im, otfpath, savedeskew, **kwargs)
    229         return decon_result, deskew_result
    230     else:
--> 231         decon_result = RL_decon(im, savedeskew=False, **kwargs)
    232         RL_cleanup()
    233         return decon_result

~\Dropbox (HMS)\Python\llspy\llspy\libcudawrapper.py in RL_decon(im, background, nIters, shift, savedeskew, rescale, **kwargs)
    257         im = im.astype(np.uint16)
    258     RL_interface(im, nx, ny, nz, decon_result, deskew_result,
--> 259                 background, rescale, savedeskew, nIters, shift)
    260
    261     if savedeskew:

OSError: [WinError -529697949] Windows Error 0xe06d7363

On the other hand, if gputools is imported after the first call to libcudaDeconv, then gputools fails to get resources

>>> import llspy
>>> import gputools  # comment out this line and it succeeds
>>> previewPath = 'basic_sample'
>>> otfdir = 'OTFs'
>>> E = llspy.LLSdir(previewPath, True)
>>> llspy.llsdir.preview(E, 0, 0, otfDir=otfdir)
output nz=84
background=0.000000
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
>>> import gputools
WARNING:gputools.config.myconfigparser | [Errno 2] No such file or directory: '~/.gputools'
>>> gputools.init_device()
 """
...choosing the best from the rest
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-45da538f8b12> in <module>()
      7 llspy.llsdir.preview(E, 0, 0, otfDir=otfdir)
      8 import gputools
----> 9 gputools.init_device()
     10

~\Anaconda3\envs\llsbuild\lib\site-packages\gputools\config\config.py in init_device(**kwargs)
     54     ....
---> 56     new_device = OCLDevice(**kwargs)
     57
     58     # just change globals if new_device is different from old

~\Anaconda3\envs\llsbuild\lib\site-packages\gputools\core\ocldevice.py in __init__(self, id_platform, id_device, use_gpu, print_info, context_properties, initCL, **kwargs)
     38                          print_info=print_info,
     39                          context_properties=context_properties,
---> 40                          **kwargs)
     41
     42     @classmethod

~\Anaconda3\envs\llsbuild\lib\site-packages\gputools\core\ocldevice.py in init_cl(self, id_platform, id_device, use_gpu, print_info, context_properties)
     92
     93         self.context = pyopencl.Context(devices=[device],
---> 94                                         properties=context_properties)
     95
     96         self.device = device

~\Anaconda3\envs\llsbuild\lib\site-packages\pyopencl\cffi_cl.py in __init__(self, devices, properties, dev_type, cache_dir)
    790             # TODO parameter order? (for clobj_list)
    791             _handle_error(_lib.create_context(_ctx, c_props,
--> 792                                               num_devices, _devices))
    793
    794         else:

~\Anaconda3\envs\llsbuild\lib\site-packages\pyopencl\cffi_cl.py in _handle_error(error)
    643     _lib.free_pointer(error.msg)
    644     _lib.free_pointer(error)
--> 645     raise e
    646
    647 # }}}

RuntimeError: clCreateContext failed: OUT_OF_RESOURCES

running CUDA DeviceQuery shows that the GPU does not seem to be running in exclusive mode:

  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

but nonetheless, the two don't work together well... Not sure where to find the equivalent setting in opencl/clinfo

doesn't seem to be a problem on mac or linux...

[tlambert03]

not a fix, but a better error message introduced in b19c9bb and 0.2.3, with the option to disable spimagine (which fixes the problem, but ... obviously... disables spimagine)

[bscott711]

Would it be possible to pin the gpuTools to one gpu, and use another for decon? This would be nice for our system since we have regularly turn uncheck our Quadro for decon since it has a smaller memory size, and just use the tesla

[tlambert03]

i'll try to look in to that. If the main issue is just that it's annoying to have to uncheck the quadro every time you start the program, it would relatively easy for me to just add that to settings that get saved automatically (so that it remembered your setting from session to session)... but i fear that the problem with gputools playing poorly with the libcudaDeconv library goes deeper than that (in the past, if I even imported them together, stuff broke on some Win10 systems), so you might still not be able to use the spimagine viewer if you're having this annoying WinError problem

that said, I don't think I ever really looked into an approach where gputools and libcudadeconv got their own GPU, so I'll try to see how hard it will be (first I need to remember some things about how it works!)

[bscott711]

Thanks. I was just thinking of a way to get the preview to work with spimagine, it's not an issue of keeping the box checked.

Using the new cudaDeconv that Lin had provided back in june, I made some modifications in a forked repo to utilize that version correctly for the bleach correction and rotation, since some of the flags had changed or were added.