binarization_dibco container fails

[masyagin1998]

Hello!
I'm trying to run your binarization_dibco.py tool in nvidia-docker container and getting following error:

F1021 20:23:03.879026 1 math_functions.cu:81] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered
*** Check failure stack trace: ***
Loading Image
Computing RD features
Concating inputs
Preprocessing
(489, 2625, 4)
Tiling input
Starting predictions for network 1/5
Progress 2%
Progress 5%
Progress 7%
Progress 10%
Progress 12%
Progress 15%
Progress 17%
Progress 20%
Progress 23%
Progress 25%
Progress 28%
Progress 30%
Progress 33%
Progress 35%
Progress 38%
Progress 41%
Progress 43%
Progress 46%
Progress 48%
Progress 51%
Progress 53%
Progress 56%
Progress 58%
Progress 61%
Progress 64%
Progress 66%
Progress 69%
Progress 71%
Progress 74%
Progress 76%
Progress 79%
Progress 82%
Progress 84%
Progress 87%
Progress 89%
Progress 92%
Progress 94%
Progress 97%
Progress 100%
Reconstructing whole image from binarized tiles
Starting predictions for network 2/5.

binarization_plm.py doesn't fail and works great :)

[ctensmeyer]

It's possible I have a bug somewhere in my Caffe code that causes the illegal access on some GPUs. What GPU are you using?

…

On Sun, Oct 21, 2018 at 2:27 PM Mikhail Masyagin ***@***.***> wrote: Hello! I'm trying to run your binarization_dibco.py tool in nvidia-docker container and getting following error: F1021 20:23:03.879026 1 math_functions.cu:81] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered *** Check failure stack trace: *** Loading Image Computing RD features Concating inputs Preprocessing (489, 2625, 4) Tiling input Starting predictions for network 1/5 Progress 2% Progress 5% Progress 7% Progress 10% Progress 12% Progress 15% Progress 17% Progress 20% Progress 23% Progress 25% Progress 28% Progress 30% Progress 33% Progress 35% Progress 38% Progress 41% Progress 43% Progress 46% Progress 48% Progress 51% Progress 53% Progress 56% Progress 58% Progress 61% Progress 64% Progress 66% Progress 69% Progress 71% Progress 74% Progress 76% Progress 79% Progress 82% Progress 84% Progress 87% Progress 89% Progress 92% Progress 94% Progress 97% Progress 100% Reconstructing whole image from binarized tiles Starting predictions for network 2/5. binarization_plm.py doesn't fail and works great :) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC2mpNX3vH_Tf-X1gEMCFp9Ri-tBlnaTks5unNiTgaJpZM4Xypq4> .

[masyagin1998]

My GPU is Nvidia 1050 Ti 4GB.

[masyagin1998]

Tried to run without GPU, but ran out of memory:
I1023 03:51:57.924324 1 net.cpp:259] Network initialization done.
I1023 03:51:57.924329 1 net.cpp:260] Memory required for data: 503054336
F1023 03:52:49.289590 1 syncedmem.hpp:33] Check failed: *ptr host allocation of size 5435817984 failed
*** Check failure stack trace: ***
Loading Image
Computing RD features
Concating inputs
Preprocessing
(722, 1087, 4)
Tiling input
Starting predictions for network 1/5

I have 8 GIBs of RAM and Intel Core I7-7700HQ. My system is Linux Mint 19.
When I started container, I had 6.5 GIBs of free RAM.

[AndyCheang]

Hello!

I am using docker to run with DIBCO2017 dataset but failure and got a similar issue.

I0410 07:38:22.774055 1 net.cpp:259] Network initialization done.
I0410 07:38:22.774060 1 net.cpp:260] Memory required for data: 503054336
F0410 07:38:22.959889 1 math_functions.cu:81] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered
*** Check failure stack trace: ***
Loading Image
Computing RD features
Concating inputs
Preprocessing
(1350, 2397, 4)
Tiling input
Starting predictions for network 1/5

[vkolagotla]

Hello,
I have the same problem as mentioned above.
I've changed the batch size to 8 (Issue shihenw/convolutional-pose-machines-release#6 (comment) it's working for most of the images(although giving washed out text) but it's still failing for some images.

Anyone found a solution?