GPU support

[sebllll]

gpumat and some other benefits would be awesome, right?

i just discovered there's new instructions on how to build and customize opencv binaries on the repo here: https://github.com/shimat/opencvsharp#how-to-customize-opencv-binaries-yourself
and even a separate repo for this purpose: https://github.com/shimat/opencv_files

i know this was not working some while ago, but maybe now is the time to try again?

[joreg]

to me it seems this is only talking about the binaries part. but my understanding is the trickier part is getting all the bindings right. some releases ago the gpumat binding were removed from the opencvsharp repo.

from my research some time ago: gpumat is the old style of doing things that is no longer further developed. the new way is relying on opencl, which i understood to be something different than gpumat. was it called umat? not so sure anymore... i might also be totally wrong on this. i had the feeling the information i found was not very clear.

for running the ML models like yolo you can simply enable the use of opencl, which on my machine curiously led to it running even slower. so i didn't investigate this further..

but ja, in general obviously it would be interesting to get gpu support for vl.opencv.

[ravazquez]

Just an update on this so it's logged somewhere:

• Initial tests were made using UMat a year or two back, showing varied results. In some operations and filters, the improvement using UMat was noticeable, especially at higher resolutions, yet in others, the performance was equal if not worse when using UMat. OpenCV's documentation does vaguely touch on the fact that not all operations are meant to be GPU optimized, so this was not a surprise, but the benefits start to appear less obvious. On top of that, it must be said that the Renderer node itself is currently implemented in the CPU, and there is no easy GPU ToImage conversion option either, which means there is a bottleneck at the end of the pipeline however you use it. This led attention to the next point:
• The newer, device-agnostic layer to OpenCV promises to be the so-called G-API, and after doing some digging in this direction, which would potentially enable direct OpenCV to-GPU and potentially even direct to-Stride interaction, it started to become clear that implementations of this G-API are hard to come by and not very well known or documented. Even less so in the C# realm. And somewhere about there is where we stopped.
• Lastly, on the UMat front again, to push this forward it is necessary to change the underlying OpenCVSharp library so that it adds support for XArray types where possible since in many cases the library's author exposed only the Mat compatible signatures, hiding potential UMat support. This would imply updating the C++ bindings and making a pull request which would then need to be merged and released by the author before it would be usable for the end-users. This can take some time. The CVImage type would also need to be deeply redesigned, maybe even dropped altogether in favour of implicit conversion support if that became a possibility.

This is not to say that all efforts are abandoned, but the road ahead is not easy or clear. The decision has for now been to wait and see where things go. Especially with GPU capable libraries like D-Lib being out there in the VL realm already.

Of course, if anyone has any ideas, by all means!