Error I/O when we try to 'Run in parallel'

[clarafischer]

It is not possible to run a process/pipeline iteration in parallel by clicking on ‘Run in parallel’. I get an error ‘I/O operation on closed file’, see below.
I've tried everything, recompiled from scratch, I've even just redone an entire brainvisa today and I still have the error.

We can launch in parallel when we create a workflow and then submit it via soma_workflow_gui.

I am using a casa-dev image version 5.3-26, master branch and cati_platform distro.
You can use it there /neurospin/cati/casa_distro/brainvisa-cati_platform-master/

ValueError: I/O operation on closed file

ValueError
Traceback (most recent call last):
  File "/casa/host/build/python/brainvisa/processing/qt4gui/neuroProcessesGUI.py", line 3099, in _run_with_soma_workflow_button
    self._run_with_soma_workflow(executionFunction=executionFunction)
  File "/casa/host/build/python/brainvisa/processing/qt4gui/neuroProcessesGUI.py", line 3070, in _run_with_soma_workflow
    SomaWorkflowWidget.brainvisa_code, serialized_process.getvalue()]
ValueError: I/O operation on closed file

[sapetnioc]

This is a code written 7 years ago that indirectly calls writeMinf() that closes the file but had been modified 10 years ago. Can someone tell me why the problem appears just now ?

[denisri]

Oh. It's possibly related to 72be634 because it's the only change which affects closing files, but this change is about logs while the error here is about soma-workflow. I'll have to dig into it.

[denisri]

This is a code written 7 years ago that indirectly calls writeMinf() that closes the file but had been modified 10 years ago. Can someone tell me why the problem appears just now ?

Ah, so it's probably in populse/soma-base@f1b5acc or populse/soma-base@558b547. That's my mistake, I'll look at it.

[denisri]

I have pushed an attempt to fix this.

[denisri]

But now when I run a Morphologist pipeline in parallel, the SPM normalization step passes, but it gets stuck in the normalization file transformation process. No idea why...

[denisri]

it's not reproducible. It sometimes hangs, and sometimes passes...

[clarafischer]

Oh super ! Thanks a lot for the rapid fix !
Weird for Morphologist, I am going to try it also to see if I have the same problem.

[denisri]

it's now stuck at another place later.
Curiously if brainvisa GUI is running in background job, it gets stopped. If I un-stop it (using "fg %1") it gets stopped again immediately. If I set it to foreground ("fg %1"), then after a while I get an error in the process about missing parameters in an internal converter.
Well I'm afraid I have completely broken something, but it will have to wait for next week.

[denisri]

I hope everything is fixed now. Can you confirm please ?

[clarafischer]

Hello, I update and test it, some things seems resolve, I could launch a 100 job in parallel and it work. But unfortunately not everything's solved, I'm losing my grip on the brainvisa interface now, I can launch one or two things and then it doesn't respond, I click on 'Run' and nothing happens, well it seems to start but doesn't return anything or return control of the GUI. Similarly, if I click on an anatomist eye, it writes in the terminal that it's starting the thing, but nothing appears in anatomist. It does this very quickly, so I can hardly use my brainvisa, which is a real handicap. Help :)

[sapetnioc]

I also have the problem using soma-env 0.0. I cannot click on more than 4 morphologist's eyes (sometime less). After that only a part of message is written on the console but nothing happens. No process is running. I start to believe that it is not related to Axon since we tried with @clarafischer to go up in time with git up to last july.