Data processing not working after first time

[Sh1ken]

How did you install LiveORC (docker, installer, self-deployed)?

Docker following the installation guide and using the ./liveorc.sh start script

I'm using v0.2.0

What's your browser and operating system?

Operating system OS X 10.15.7
Javascript enabled Yes
Cookies enabled Yes
Flash version Not installed
Java version Not installed
Websockets supported Yes
WebGL supported Yes

Please note that I also tried the setup on a Windows 11 machine using WSL and got exactly the same error.

What is the problem?

After running the example with the sample data, I tried to run it a second time deleting the initial video and then running it again. I got the following trace:

liveorc_worker-1  | [2025-04-23 15:45:36,823: ERROR/ForkPoolWorker-6] Error in processing of subtask f8f8790b-debd-4d4a-a0d6-f65f64d07c97. Reason: Temporary file /tmp/nodeorc/videos/1/20250423/OUTPUT/plot_quiver.jpg was not created by subtask
liveorc_worker-1  | [2025-04-23 15:45:36,978: ERROR/ForkPoolWorker-6] Task api.tasks.run_nodeorc[1969d5cb-6d12-43f3-ad17-70f726f86f30] raised unexpected: Exception('Error in processing of subtask f8f8790b-debd-4d4a-a0d6-f65f64d07c97. Reason: Temporary file /tmp/nodeorc/videos/1/20250423/OUTPUT/plot_quiver.jpg was not created by subtask')
liveorc_worker-1  | Traceback (most recent call last):
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/task.py", line 69, in execute
liveorc_worker-1  |     self.execute_subtasks(tmp, timestamp=self.timestamp)
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/task.py", line 115, in execute_subtasks
liveorc_worker-1  |     subtask.execute(
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/subtask.py", line 49, in execute
liveorc_worker-1  |     self.upload_outputs(task.storage, tmp)
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/subtask.py", line 119, in upload_outputs
liveorc_worker-1  |     raise FileNotFoundError(f"Temporary file {tmp_file} was not created by subtask")
liveorc_worker-1  | FileNotFoundError: Temporary file /tmp/nodeorc/videos/1/20250423/OUTPUT/plot_quiver.jpg was not created by subtask
liveorc_worker-1  |
liveorc_worker-1  | During handling of the above exception, another exception occurred:
liveorc_worker-1  |
liveorc_worker-1  | Traceback (most recent call last):
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
liveorc_worker-1  |     R = retval = fun(*args, **kwargs)
liveorc_worker-1  |                  ^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
liveorc_worker-1  |     return self.run(*args, **kwargs)
liveorc_worker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/liveorc/api/tasks.py", line 41, in run_nodeorc
liveorc_worker-1  |     task.execute(tmp="/tmp/nodeorc", keep_src=True)  #os.path.dirname(trg))
liveorc_worker-1  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/task.py", line 76, in execute
liveorc_worker-1  |     raise Exception(msg)
liveorc_worker-1  | Exception: Error in processing of subtask f8f8790b-debd-4d4a-a0d6-f65f64d07c97. Reason: Temporary file /tmp/nodeorc/videos/1/20250423/OUTPUT/plot_quiver.jpg was not created by subtask
liveorc_worker-1  | [2025-04-23 15:45:36,997: ERROR/ForkPoolWorker-6] Signal handler <function task_postrun_handler at 0x7fff9201da80> raised: <HTTPError 404: 'Not Found'>
liveorc_worker-1  | Traceback (most recent call last):
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/utils/dispatch/signal.py", line 276, in send
liveorc_worker-1  |     response = receiver(signal=self, sender=sender, **named)
liveorc_worker-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/liveorc/api/tasks.py", line 65, in task_postrun_handler
liveorc_worker-1  |     add_frame_to_model(
liveorc_worker-1  |   File "/liveorc/api/models/video.py", line 60, in add_frame_to_model
liveorc_worker-1  |     urllib.request.urlopen(
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 216, in urlopen
liveorc_worker-1  |     return opener.open(url, data, timeout)
liveorc_worker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 525, in open
liveorc_worker-1  |     response = meth(req, response)
liveorc_worker-1  |                ^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 634, in http_response
liveorc_worker-1  |     response = self.parent.error(
liveorc_worker-1  |                ^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 563, in error
liveorc_worker-1  |     return self._call_chain(*args)
liveorc_worker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 496, in _call_chain
liveorc_worker-1  |     result = func(*args)
liveorc_worker-1  |              ^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 643, in http_error_default
liveorc_worker-1  |     raise HTTPError(req.full_url, code, msg, hdrs, fp)
liveorc_worker-1  | urllib.error.HTTPError: HTTP Error 404: Not Found

I also tried to run it with my own data and got a pretty similar error.

The behavior is sketchy because it works just fine the first time, I got exactly the same output as you show in the example but then it throws this error again and again.

How can we reproduce this?

Try using the sample data provided, process the video.
Delete said video and try running the same process again.

Thanks for all the hard work that you are putting into the project. Looks amazing!

[hcwinsemius]

Hi @Sh1ken - sorry for an extremely late answer. I for some reason have not received (or it went into a spam box??) a notification of your issue, and just stumbled on it whilst working on a number of updates.

I have just tried to reproduce your issue but am not able to do so. What I did:

• follow the example with uploading recipe, profile and making a camera config
• upload the sample video
• Make a new time series object with water level
• Hit the "play" button, and wait until processing is done. Check results.
• Delete the video (not the time series)
• Upload the video again (time series gets linked automatically)
• Click "play" button again
• wait until done and check results.

This worked fine. Also the logs did not show any errors. Can you perhaps let me know if you see any differences between what I did and what you did? Also can you copy the entire log?

Thanks for posting, and again apologies.

[Sh1ken]

Hey @hcwinsemius, no worries. I see by the commit history that you're (at least currently) almost single-handedly pushing this project forward so I don't want to put any more work over your shoulders, haha.

The steps that you described are exactly the same that I did last time when I tried to recreate this error. Now I downloaded the new version of LiveORC, clear all cached imgs and containers on my side and ran the project again. I followed the steps that you describe until the first "Check results" but I got and error, similar to the one that I had before but not quite the same.

liveorc_worker-1  | [2025-07-10 21:46:16,468: ERROR/ForkPoolWorker-6] Error in processing of subtask 22c6b718-187a-43ca-bedf-3dad30688980. Reason: Temporary file /tmp/nodeorc/videos/1/20250710/OUTPUT/transect_transect_1.nc was not created by subtask
liveorc_worker-1  | [2025-07-10 21:46:16,764: ERROR/ForkPoolWorker-6] Task api.tasks.run_nodeorc[5cb6ca05-11f9-4163-be9d-419ad51f9dae] raised unexpected: Exception('Error in processing of subtask 22c6b718-187a-43ca-bedf-3dad30688980. Reason: Temporary file /tmp/nodeorc/videos/1/20250710/OUTPUT/transect_transect_1.nc was not created by subtask')
liveorc_worker-1  | Traceback (most recent call last):
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/task.py", line 69, in execute
liveorc_worker-1  |     self.execute_subtasks(tmp, timestamp=self.timestamp)
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/task.py", line 115, in execute_subtasks
liveorc_worker-1  |     subtask.execute(
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/subtask.py", line 49, in execute
liveorc_worker-1  |     self.upload_outputs(task.storage, tmp)
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/subtask.py", line 119, in upload_outputs
liveorc_worker-1  |     raise FileNotFoundError(f"Temporary file {tmp_file} was not created by subtask")
liveorc_worker-1  | FileNotFoundError: Temporary file /tmp/nodeorc/videos/1/20250710/OUTPUT/transect_transect_1.nc was not created by subtask
liveorc_worker-1  |
liveorc_worker-1  | During handling of the above exception, another exception occurred:
liveorc_worker-1  |
liveorc_worker-1  | Traceback (most recent call last):
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
liveorc_worker-1  |     R = retval = fun(*args, **kwargs)
liveorc_worker-1  |                  ^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
liveorc_worker-1  |     return self.run(*args, **kwargs)
liveorc_worker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/liveorc/api/tasks.py", line 41, in run_nodeorc
liveorc_worker-1  |     task.execute(tmp="/tmp/nodeorc", keep_src=True)  #os.path.dirname(trg))
liveorc_worker-1  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/task.py", line 76, in execute
liveorc_worker-1  |     raise Exception(msg)
liveorc_worker-1  | Exception: Error in processing of subtask 22c6b718-187a-43ca-bedf-3dad30688980. Reason: Temporary file /tmp/nodeorc/videos/1/20250710/OUTPUT/transect_transect_1.nc was not created by subtask
liveorc_worker-1  | [2025-07-10 21:46:16,795: ERROR/ForkPoolWorker-6] Signal handler <function task_postrun_handler at 0x7fff9201da80> raised: <HTTPError 404: 'Not Found'>
liveorc_worker-1  | Traceback (most recent call last):
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/utils/dispatch/signal.py", line 276, in send
liveorc_worker-1  |     response = receiver(signal=self, sender=sender, **named)
liveorc_worker-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/liveorc/api/tasks.py", line 65, in task_postrun_handler
liveorc_worker-1  |     add_frame_to_model(
liveorc_worker-1  |   File "/liveorc/api/models/video.py", line 60, in add_frame_to_model
liveorc_worker-1  |     urllib.request.urlopen(
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 216, in urlopen
liveorc_worker-1  |     return opener.open(url, data, timeout)
liveorc_worker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 525, in open
liveorc_worker-1  |     response = meth(req, response)
liveorc_worker-1  |                ^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 634, in http_response
liveorc_worker-1  |     response = self.parent.error(
liveorc_worker-1  |                ^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 563, in error
liveorc_worker-1  |     return self._call_chain(*args)
liveorc_worker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 496, in _call_chain
liveorc_worker-1  |     result = func(*args)
liveorc_worker-1  |              ^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 643, in http_error_default
liveorc_worker-1  |     raise HTTPError(req.full_url, code, msg, hdrs, fp)
liveorc_worker-1  | urllib.error.HTTPError: HTTP Error 404: Not Found

I actually don't know if there's anything worthwhile in the other parts of the logs that can help debug the problem.

rabbitmq          | 2025-07-10 21:44:25.963026+00:00 [info] <0.1071.0> accepting AMQP connection <0.1071.0> (172.20.0.5:54232 -> 172.20.0.4:5672)
rabbitmq          | 2025-07-10 21:44:25.982449+00:00 [info] <0.1071.0> connection <0.1071.0> (172.20.0.5:54232 -> 172.20.0.4:5672): user 'admin' authenticated and granted access to vhost '/'
liveorc_webapp    |  - - [10/Jul/2025:21:44:26 +0000] "GET /admin/api/video/2/actions/toolfunc/ HTTP/1.0" 302 0 "http://localhost:8000/admin/api/video/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
liveorc_webapp    |  - - [10/Jul/2025:21:44:26 +0000] "GET /admin/api/video HTTP/1.0" 301 0 "http://localhost:8000/admin/api/video/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
liveorc_worker-1  | [2025-07-10 21:44:26,118: WARNING/ForkPoolWorker-6] I received a task with task id: 22c6b718-187a-43ca-bedf-3dad30688980
liveorc_worker-1  | [2025-07-10 21:44:26,120: WARNING/ForkPoolWorker-6] Copying video file to /tmp/nodeorc/videos/1/20250710/20250710T214419.mp4
liveorc_webapp    |  - - [10/Jul/2025:21:44:26 +0000] "GET /admin/api/video/ HTTP/1.0" 200 40235 "http://localhost:8000/admin/api/video/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
liveorc_webapp    |  - - [10/Jul/2025:21:44:26 +0000] "GET /admin/jsi18n/ HTTP/1.0" 200 3342 "http://localhost:8000/admin/api/video/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
liveorc_webapp    |  - - [10/Jul/2025:21:44:26 +0000] "GET /api/site/1/video/2/thumbnail/ HTTP/1.0" 200 1083 "http://localhost:8000/admin/api/video/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"

Those are the lines that are just above the ones that I previously posted. Is there anything else I can do so you can try to replicate the problem? As I originally said, this behaviour was the same on Windows and OSX so I don't think that it has to do with my particular setup.

[hcwinsemius]

Thanks for trying this all out again. I can see that the task does not return files, which suggests that issuing the task is going fine, but on the node orc side, things are not working as they should. Just to confirm: this happens only the second time you run the video correct?

I cannot reproduce it for some reason. Could you however try the following:

• delete the video again
• also delete all time series objects (see menu item Time series, this is a new step)
• Then add the video again, check if it does not receive a time series object link
• Then add a time series object with it
• Then rerun

One of the issues is that no log of the processing is returned to the user. We are working on an entirely new back end (ORC API) to better manage jobs. I will include an issue there, to add log file availability to it. That should make it easier to see where the job fails on the back-end side.

[Sh1ken]

Hey @hcwinsemius

Sadly, today I don't bring good news this time neither. 😔
I followed the steps that you listed and got this error now.

liveorc_worker-1  | [2025-07-15 15:27:25,122: ERROR/ForkPoolWorker-6] Error in processing of subtask 40a0014d-1ad2-4d13-bb36-7e71d669b1f4. Reason: Temporary file /tmp/nodeorc/videos/1/20250715/OUTPUT/plot_quiver.jpg was not created by subtask
liveorc_worker-1  | [2025-07-15 15:27:25,488: ERROR/ForkPoolWorker-6] Task api.tasks.run_nodeorc[5febe358-4e8f-4cf1-b531-335681d6deb9] raised unexpected: Exception('Error in processing of subtask 40a0014d-1ad2-4d13-bb36-7e71d669b1f4. Reason: Temporary file /tmp/nodeorc/videos/1/20250715/OUTPUT/plot_quiver.jpg was not created by subtask')
liveorc_worker-1  | Traceback (most recent call last):
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/task.py", line 69, in execute
liveorc_worker-1  |     self.execute_subtasks(tmp, timestamp=self.timestamp)
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/task.py", line 115, in execute_subtasks
liveorc_worker-1  |     subtask.execute(
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/subtask.py", line 49, in execute
liveorc_worker-1  |     self.upload_outputs(task.storage, tmp)
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/subtask.py", line 119, in upload_outputs
liveorc_worker-1  |     raise FileNotFoundError(f"Temporary file {tmp_file} was not created by subtask")
liveorc_worker-1  | FileNotFoundError: Temporary file /tmp/nodeorc/videos/1/20250715/OUTPUT/plot_quiver.jpg was not created by subtask
liveorc_worker-1  |
liveorc_worker-1  | During handling of the above exception, another exception occurred:
liveorc_worker-1  |
liveorc_worker-1  | Traceback (most recent call last):
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
liveorc_worker-1  |     R = retval = fun(*args, **kwargs)
liveorc_worker-1  |                  ^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
liveorc_worker-1  |     return self.run(*args, **kwargs)
liveorc_worker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/liveorc/api/tasks.py", line 41, in run_nodeorc
liveorc_worker-1  |     task.execute(tmp="/tmp/nodeorc", keep_src=True)  #os.path.dirname(trg))
liveorc_worker-1  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/nodeorc/models/task.py", line 76, in execute
liveorc_worker-1  |     raise Exception(msg)
liveorc_worker-1  | Exception: Error in processing of subtask 40a0014d-1ad2-4d13-bb36-7e71d669b1f4. Reason: Temporary file /tmp/nodeorc/videos/1/20250715/OUTPUT/plot_quiver.jpg was not created by subtask
liveorc_worker-1  | [2025-07-15 15:27:25,717: ERROR/ForkPoolWorker-6] Signal handler <function task_postrun_handler at 0x7fff91e29a80> raised: <HTTPError 404: 'Not Found'>
liveorc_worker-1  | Traceback (most recent call last):
liveorc_worker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/utils/dispatch/signal.py", line 276, in send
liveorc_worker-1  |     response = receiver(signal=self, sender=sender, **named)
liveorc_worker-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/liveorc/api/tasks.py", line 65, in task_postrun_handler
liveorc_worker-1  |     add_frame_to_model(
liveorc_worker-1  |   File "/liveorc/api/models/video.py", line 60, in add_frame_to_model
liveorc_worker-1  |     urllib.request.urlopen(
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 216, in urlopen
liveorc_worker-1  |     return opener.open(url, data, timeout)
liveorc_worker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 525, in open
liveorc_worker-1  |     response = meth(req, response)
liveorc_worker-1  |                ^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 634, in http_response
liveorc_worker-1  |     response = self.parent.error(
liveorc_worker-1  |                ^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 563, in error
liveorc_worker-1  |     return self._call_chain(*args)
liveorc_worker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 496, in _call_chain
liveorc_worker-1  |     result = func(*args)
liveorc_worker-1  |              ^^^^^^^^^^^
liveorc_worker-1  |   File "/usr/local/lib/python3.11/urllib/request.py", line 643, in http_error_default
liveorc_worker-1  |     raise HTTPError(req.full_url, code, msg, hdrs, fp)
liveorc_worker-1  | urllib.error.HTTPError: HTTP Error 404: Not Found

Answering your question: When I first reported this problem, I was able to run the entire process one time and then the second one I started to get errors. But now, with the current version, I haven't been able to run the process even once.

Anyway, I think that with the new backend that you mentioned we'll be able to debug this problem with greater detail for sure. Let me know when it's ready or if there's anything that I could do/test in the meantime.

Cheers!

[hcwinsemius]

Hmm. How about a short videocon after my holidays? I am back 11 Aug.

[Sh1ken]

For sure! I'll drop you an email right away. Thanks!