ffmpeg hanging on motion capture (sometimes)

[Zopiac]

Sorry to not have a fully understood trigger for this issue, but I'm having problems with motion triggered video capture not saving, seemingly because there's an ffmpeg thread 'stuck' indefinitely:

├─ 2240 ffmpeg -loglevel warning -y -use_wallclock_as_timestamps 1 -thread_queue_size 64 -i - -c:v copy /home/zopiac/raspi-cam-srv/raspiCamSrv/static/events/2025-04-13T11-19-31.mp4

Latest errors in systemd log at the time were:

Apr 13 11:19:33 zhkpi02w3 flask[2240]: [mp4 @ 0xaa2750] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
Apr 13 11:19:33 zhkpi02w3 flask[2240]: [mp4 @ 0xaa2750] Non-monotonous DTS in output stream 0:0; previous: 211642, current: 172323; changing to 211643. This may result in incorrect timestamps in the output file.

but these are also present for other encodes which complete successfully. It had gone hours without failing, recording dozens of motion events, before this one popped up. Restarted the service and it happened again within half an hour. Six hours later and I'm still waiting for it to repeat itself.

On the other hand, yesterday I was messing around with different settings and it froze like this nearly every single time motion triggered -- I think it had to do with the sensor mode(s) I was using. Today nothing has changed but it "randomly" hangs on some captures. Maybe resolution or bitrate settings can cause some pipeline to choke with varying frequency?

Killing the ffmpeg process doesn't allow the server to continue detecting motion; restarting the service seems to be the best bet. A rudimentary workaround may be to routinely check for a running ffmpeg process, waiting a minute, and then restarting the service if it's that process is still running. I can try and figure out more about the issue if I'm given somewhere/something to look for, though.

I am currently using a Raspberry Pi 3 with OV5647 camera sensor (640x480 lores mode) and Raspbian Bookworm on kernel 6.12.20. I get memory issues on a Zero 2 but can test on a Pi 5 if it would help... with no guarantee it'll trigger within a day or even a week.

[signag]

Hi,
thanks for posting this.

I have checked the code and I have seen that for Pi 1, 2, 3 the Video configuration was used for videos recorded during motion capturing and not, as for Pi Zero and Pi 4 the Live View configuration. I do not have these models and I cannot test these. I only have Pi Zeros, Pi 4 and Pi 5.

With Pi 4, I did not find any issues.
I also do not see memory issues on Pi Zero 2 W with bookworm.
However, on a Pi Zero 2, I see similar log messages as you have reported. But these are warnings and would need to be handled in Picamera2 and not in raspiCamSrv. I do not think that they are related to your ffmpeg issues.

So,. probably, in your case with a Pi 3, the issues were seen because the Video Configuration with the higher resolution and memory requirements were used. Probably, you can check the videos to see which resolution they have.

I have fixed this in V3.5.1.

If you update to this version, you could check if the issue is still there.

On the other hand ....
If you are using lores stream in the Live View Configuration, you will certainly see a low live view quality, due to the YUV format. With version V3.5.1 you will see the same bad quality in the videos.

On my Pi Zeros and Pi 4, I usually use the "main" stream with low resolution and the RGB888 Stream Format for the Live View configuration, which gives much better quality.

[Zopiac]

Thanks for the response! I didn't think that the warnings in log were related to the problem -- I half posted them to simply show that there was nothing more troublesome than that being reported.

The video format being saved on my Pi 3 has been a 640 x 480 YUV video at <500kb/s bitrate (mediainfo reports 240kb/s with 475bk/s nominal bit rate). Interestingly (and this is probably unrelated), it has consistently recorded 8.9s when 10 seconds is specified as video record length (the default). Since my last post I upped it to 15 seconds and upon checking it now, it's saving 13.8s of video... except for once when it went on for almost six minutes?? It didn't actually record that much footage (the filesize is a quarter that of the other videos, with an average 0.090 FPS framerate) as it clearly stalled for quite some time. I wonder if this happens to be an instance where ffmpeg was able to recover, eventually.

I just cloned raspiCamSrv fresh, set up motion capture again, but this time selected 'main' for Stream on the Config tab. The very first motion trigger caused ffmpeg to lock up entirely. Even lowering the resolution to 320x240 didn't help. Restarting the service and switching back to 'lores' fixed this. So at least that seems like a decent way to trigger the lockup on demand... although I think the lores stream is making consistent motion detection difficult (the mjpeg compression seems to trigger even a Mean Square Threshold of 20).

Also... I just noticed that I've only been adjusting the resolution in the Live View tab of Config. The Video tab has been, I presume, 640 x 480 from 'main' stream this entire time, given that I haven't changed that. Adjusting live/photo/video resolutions to mode_1 (1296 x 972) but keeping Live View on 'lores' works fine, but at this resolution on 'main' it's even more perplexing... the ffmpeg process exits (unsuccessfully -- no video is saved), but it seems to crash out the camera entirely with errors:

Apr 14 15:29:15 zhkpi02w3 flask[5871]: [0:29:32.567006267] [6840] ERROR Camera camera.cpp:711 Camera in Available state trying configure() requiring state between Acquired and Configured
Apr 14 15:29:15 zhkpi02w3 flask[5871]: [2025-04-14 15:29:15,817] ERROR in camera_pi: Thread 1453261824: CameraController.requestStart - Error starting camera: Failed to configure camera: Permission denied
Apr 14 15:29:15 zhkpi02w3 flask[5871]: [0:29:32.584570487] [6840] ERROR Camera camera.cpp:711 Camera in Available state trying configure() requiring state between Acquired and Configured
Apr 14 15:29:15 zhkpi02w3 flask[5871]: [2025-04-14 15:29:15,831] ERROR in camera_pi: Thread 1453261824: CameraController.requestStart - Error starting camera: Failed to configure camera: Permission denied
Apr 14 15:29:20 zhkpi02w3 flask[5871]: [0:29:37.531012838] [6891] ERROR Camera camera.cpp:690 Camera in Available state trying start() requiring state Configured
Apr 14 15:29:20 zhkpi02w3 flask[5871]: [2025-04-14 15:29:20,778] ERROR in camera_pi: Thread 1461654528: CameraController.requestStart - Error starting camera: Failed to start camera: Permission denied
Apr 14 15:29:20 zhkpi02w3 flask[5871]: [0:29:37.545524574] [6891] ERROR Camera camera.cpp:711 Camera in Available state trying configure() requiring state between Acquired and Configured
Apr 14 15:29:20 zhkpi02w3 flask[5871]: [2025-04-14 15:29:20,792] ERROR in camera_pi: Thread 1461654528: CameraController.requestStart - Error starting camera: Failed to configure camera: Permission denied

(Ignore the hostname -- I meant for this to run on a Zero 2 W but after the aforementioned memory issues I moved the microSD to my Pi 3) Restarting the service (thus reloading my 640x480 config with Live View on lores) fixes that error.

Speaking of the Zero 2 W though, I just swapped the card back into that and hooked it up to the camera and it seems to now be working fine. No clue why, as even libcamera-hello wouldn't work with it the other day ([1454] ERROR V4L2 v4l2_videodevice.cpp:1241 /dev/video0[12:cap]: Unable to request 4 buffers: Cannot allocate memory / [1454] ERROR RPI pipeline_base.cpp:681 Failed to allocate buffers). Gotta love gremlins. So now I have the server running on that, but it'll be a few days before I'll be accessing it again. I'll give an update then, or whenever I have free time after that point.

[signag]

Hi,
I just want to give an update, especially for Pi Zero and Pi Zero 2.
Some time ago, I had moved to using bookworm on these devices because I got the impression that this seemed to be stable.

However, there were issues with the bad YUV MJPEG stream quality and also with video recording, not only during motion capturing.

Today, I did some testing with a Pi Zero running on bullseye.
And this did not show the issues observed with bookworm, Neither did the MJPEG stream with YUV format show quality artifacts nor did I have problems with video recording.

I am attaching some screenshots:

Within about an hour of operation with active motion capturing, the system has registered about 230 events and recorded the same number of videos:

But also here, I see the message

Apr 19 15:37:02 raspi04 flask[4573]: [mp4 @ 0x20dc500] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly

This has also been reported in Picamera2 Issue: ffmpeg exception 1053, however without a final solution, as far as I can see.

As you mentioned memory issues on Pi Zero 2, I had a look at my performance monitoring for the Pi Zero W during the test:

With memory usage of ~30%, I do not see any issues.

However, with an average CPU utilization of ~70% and CPU temperature of ~60 °C this is probably not the ideal device for 24/7 motion detection.

Summarizing, I would say that bullseye seems to be preferable over bookworm, at least on Pi Zero, and if video recording is required.

I have a Pi Zero 2 with bookworm recording a photo series for several month now, which works absolutely fine.

[Zopiac]

The only 'memory' issue I had with the Zero 2 was the aforementioned error on attempting to initialise the camera, rather than noticing any actual high memory usage. It seemed to fail out without even attempting, best as I could tell.

At any rate, either that resolved itself or I fixed it in my sleep somehow. The Zero 2 W had been capturing photo and video (mostly) successfully for four days straight (and I mean straight, as the background noise was easily triggering motion capture so it captured 8GB of mostly useless data, but that's just down to tweaking the threshold properly, I hope).

However, it managed to get hot enough to soften the 3D printed mounting bracket it was attached to, and so I decided to try limiting it to 600MHz in an attempt to manage heat, but this actually almost immediately caused the Zero 2 W to start exhibiting issues much like my Pi 3 did, as ffmpeg would hang indefinitely, requiring service restart. Could it be from not finishing encoding the video (or writing the file) before another motion detection occurs? Hopefully I'll do some testing but it's a bit late and I have a busy day coming up.

What I do know is that switching Live View to main stream seems to guarantee that the next recording will hang.

Info screen for posterity:

[egge12]

Hi

I also have issues with motion capture on a PiZero2W - maybe also related to ffmpeg?

I'm running raspiCamSrv V3.5.2 on a Pi5 with camera imx477 and also on a PiZero2W with camera ov5647 - both on latest version of bookworm 64bit. Live view works well for both setups.

However it looks like that the PiZero2W does not capture pictures and also no videos. When trying motion detection an error message is displayed (see screenshot below).

[signag]

Hi egge12,

Since the reported error originates from OpenCV, it is very likely that you are using one of the advanced motion tracking algorithms like "Background Subtraction", because these are the only ones using OpenCV.
Your issue is related to the image format of the Live View which requires to be YUV for Pi models < 5 and which cannot be processed within OpenCV.

On a Pi Zero 2W, I could successfully capture motion with the following configuration for the Live View:

In order to avoid the ffmpeg issue, I have chosen to activate Video with Bounding Boxes because in this case, the video is not recorded through the ffmpeg encoder but image by image through the VideoWriter of OpenCV. That is why the Info screen does not show ffmpeg process activity.

Here are the events captured with this method within the last half hour:

Related to ffmpeg encoding, my experience is better with Bullseye compared to Bookworm.
However, then you are restricted to the Mean Square Algorithm, because OpenCV cannot be used on such systems.

[egge12]

Hi

thanks for quick reply... On PiZero2W with bookworm I used "MeanSquareDiff"... and you are right, with the standard settings in LiveView I got very blured image. I did similar ajustments as you mentioned and got at least a good live view. However no motion detection...

Luckily I had another SD-card with bullseye 32bit. For testing I booted this on my PiZero2W and installed raspi-cam-srv. There everything is working as expected. However as you described only "MeanSquareDiff" available... I will continue testing and if this fullfills my needs I will stay with bullseye.

On the other side, it would be a pitty, it your nice camera-application will not work properly on PiZero2W and Bookworm... if you want I can continue testing on this.

[signag]

It is very likely that motion tracking on Pi Zero with Bookworm does not work because of the ffmpeg issue.
You could try "Background Subtraction" instead of "Mean Square Diff" and keep *Video with Bounding Boxes" checked.

If "Background Subtraction" is not available as option, you need to install OpenCV, numpy and matplotlib (see RaspiCamSrv Installation step 10). This is possible when running Bookworm on a Pi Zero. With Bullseye it is not.

You can also check the events log file at respiCamSrv/static/events/_events.log
Probably, you will see error messages for video recording.

[egge12]

I can confirm, that Bullseye does not support advanced motion capturing...

in the meantime I setup a new SD-card with Bookworm 32bit (headless) on my PiZero2W and installed raspi-cam-srv V3.5.2 (also ffmpeg; python3-opencv; numpy; matplotlib; flask-jwt-extended). Unfortunately still no advanced motion capturing - only "Mean Squar Diff".
In you last post you mentioned to install OpenCV - is this same as python3-opencv? Or might it be related to 32bit version or that I installed the headless Bookworm version (meaning without Desktop).

[signag]

In you last post you mentioned to install OpenCV - is this same as python3-opencv?

Yes, thats the package to be installed (see RaspiCamSrv Installation step 10).

Whether it could be imported successfully can be seen in dialog Settings/Parameters.
If all required packages could be loaded, Ext. Motion Detection supported will be checked. If not, it will be indicated which package is missing.
If it is checked, you should be offered the extended algorithms in Trigger/Motion.
Please verify.

Not sure whether 32bit OS would prevent one of the reqired packages to be successfully installed.
I always test with the OS recommended by Imager, which is typically the full 64bit OS version.

[egge12]

yes, I can confirm that everything was installed according to RaspiCamSrv Installation, but there was no extended motion detection - only mean square diff.

now I followed your recommendation and installed on PiZero2W recommended OS by imager (V1.8.5), which is Bookworm 64bit with desktop. Installation of raspi-cam-srv (exactly as described in instructions) went smoothly and application runs well. Advanced motion detection is available, but testing mode is not working at all. I put some screenshot below... Event.log is empty, no failure indications... and it also looks like that enough memory is available... On my Pi5 RaspiCamSrv runs without any issues.

[signag]

Hi,

I tested this with the same kernel and Bookworm version as you have reported, however with a imx708_wide_noir instead of your Waveshare ov5647.

The videos of the Motion Capturing test showed up as they should.
However there is a little time delay (about 1-2 sec), which depends on the algorithm used.

I think that this may be attributed to the low performance of these devices which also shows up in the very low frame rate achievable in this mode:

May be, there is also a dependency on the camera. Probably drivers for compatible cameras are not as efficient as those for original cameras.

To be honest, I find it already remarkable that this works at all on these small devices.
I am seeing high CPU utilization of > 95% and CPU temperatures of 80°C and more during these tests.

[egge12]

thanks a lot... in the meantime I did some further testing and also get the impression, that my issues are related to low performance of Raspi PiZero2W - and not due to ffmpeg issues.

With the original raspi HQ cam (imx477) I got very blured image in live view. With your recommended settings this is solved, however motion detection (test modus and real) the advanced features still not work - I still get the same failure as reported before. Very strange - on your PiZero it seem to work without this failure.

As conclusion, I think Raspi-Cam-Srv runs well on PiZero 2W, as long as the right settings are used and motion detection only used in "mean square diff". Unfortunately controlling sensitivity is here quite hard and I get lots of faults (false negative and false positive - depending on threshold). Since I'm observing outside in nature a masking function could improve this - I alredy have requested this here

On the other hand, I can confirm, that everything runs very well on my Pi5 (also with both cameras imx477 and ov5647). Motion detection with advanced features works very well and gives very satisfying results.

thanks for all your support :-)