ffmpeg: Add V4L2 Request API decoders #179
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kwiboo
force-pushed
the
ffmpeg-v4l2request
branch
from
549198d to
758cb69
Compare
Kwiboo
added a commit
to Kwiboo/FFmpeg
that referenced
this pull request
Aug 6, 2024
This is a follow up to a very old series from April 2019 [1] and
December 2020 [2], adding V4L2 Request API hwaccels for stateless
decoding of MPEG2, H.264 and HEVC.
These hwaccels has in one form or another been used in LibreELEC
community, nightly and release images since Dec 20th 2018.
HISTORY
The initial v4l2-request hwaccel code was mainly created as a proof of
concept at the end of 2018 by me and Jernej Skrabec. Back when Bootlin's
crowdfunding campaign for upstream Linux kernel driver for Allwinner VPU
had started to bear fruit.
At the time we had very little knowledge on how to properly interact
with any of the V4L2 APIs, and the existing V4L2 M2M code in FFmpeg
seemed very complex to start working with, so we started from scratch.
(That is the main reason why these hwaccels still does not use any of
the existing V4L2 M2M code in FFmpeg.)
The hwaccels had one major limitation, it waited on the kernel to
complete decoding before continuing with next request. Due to hwaccels
still was able to decode up to 4k 50-60fps on Allwinner and Rockchip
boards, time was never spent trying to improve this limitation.
The initial version of these hwaccels was merged as patches into
LibreELEC in April 2019 [3], with my FFmpeg tree as main source for any
update of FFmpeg v4l2-request patches included in LibreELEC.
A RFC was sent in April 2019 [1], with very little feedback and the
required kernel headers not being merged until a year later there was
never any new RFC revision submitted.
After the release of the Raspberry Pi 4 in June 2019 there was hope that
RPi folks would rework and/or improve our proof-of-concept to something
better that could be upstreamed, as history has show this did not happen.
And instead we have just ended up with a second FFmpeg v4l2-request
implementation that has only really been used for HEVC decoding on RPi.
Due to personal reasons I took a long break from doing open-source
contributions during ~2020-2023. During this time Jernej's FFmpeg tree
became the main source for FFmpeg v4l2-request patches in LibreELEC [4].
In December 2020 a v1 of this series was sent [2], at that time kernel
headers for H.264 had been merged into Linux kernel, and I cannot
remember why it never went any further.
Last update to the v4l2-request patches in LibreELEC was done by me in
November 2023 [5].
Along the way there has been noteworthy contributions including from
Boris Brezillon, Ezequiel Garcia, Alex Bee and Benjamin Gaignard.
PRESENT DAY
The version submitted in this series have seen major refactoring to the
common code, to make it more ready for upstreaming and also a rework of
how buffers and requests are handled.
The limitation of waiting for decoding to complete has been removed and
now multiple pending requests can be in-flight at the same time. This
was a requirement to better support HEVC decoding on RPi.
The older versions also incorrectly required use of a DRM hwdevice,
however V4L2 decoding has nothing to do with the DRM subsystem in Linux.
Instead in this version a new V4L2 Request API hwdevice has been added.
Thanks to this it is now possible to specify what media device to use
for decoding, in case multiple decoders exists on a system. E.g. using a
-init_hw_device v4l2request:/dev/media1 parameter.
This version only add support for MPEG2, H264 and HEVC. Support for VP8,
VP9 and AV1 is planned and be added in next revision or in a follow up
series.
HOW TO USE
To use the V4L2 Request API hwaccels you must build FFmpeg on a system
with recent Linux kernel headers, v6.0+. It also requires libdrm and
libudev to successfully build.
This can then be runtime tested on multiple Allwinner and Rockchip
devices. To runtime test this on a RPi 4 or 5 you should use latest
rpi-6.6.y kernel.
ffmpeg -hwaccel v4l2request -hwaccel_output_format drm_prime \
-i <input-path> -map 0:v -f null -
This series has been tested with cedrus driver on Allwinner H6, hantro
and rkvdec driver on Rockchip RK3399, and rpivid driver on RPi 4/5.
PRs have also been opened for Kodi and mpv to assist with the transition
from DRM to a new V4L2REQUEST hwdevice type.
- Kodi: xbmc/xbmc#25467
- mpv: mpv-player/mpv#14511
With those PRs applied it should be possible to playback video using
kodi-gbm or mpv, see the PRs above for more details.
It should also be possible to run fluster test suites with following PR:
- fluster: fluendo/fluster#179
I am expecting that there will be a new revision adding VP8, VP9 and AV1
support in a week or two.
Please get back with any type of feedback!
A copy of this series can also be found at [6].
[1] https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2019-April/242316.html
[2] https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2020-December/273579.html
[3] LibreELEC/LibreELEC.tv#3405
[4] https://github.com/LibreELEC/LibreELEC.tv/commits/master/packages/multimedia/ffmpeg/patches/v4l2-request
[5] LibreELEC/LibreELEC.tv#8356
[6] https://github.com/Kwiboo/FFmpeg/commits/v4l2request-2024-v2/
Signed-off-by: Jonas Karlman <jonas@kwiboo.se>
Contributor
|
@Kwiboo did you have any luck getting your FFmpeg changes accepted and merged? |
Kwiboo
force-pushed
the
ffmpeg-v4l2request
branch
from
758cb69 to
d2426fc
Compare
mdimopoulos
force-pushed
the
ffmpeg-v4l2request
branch
from
d2426fc to
667bbd0
Compare
Contributor
|
The failure is caused because of the commit message. |
mdimopoulos
force-pushed
the
ffmpeg-v4l2request
branch
from
667bbd0 to
7736727
Compare
azure-pipelines bot
pushed a commit
to ffstaging/FFpatches
that referenced
this pull request
Nov 5, 2025
… #20847) PR #20847 opened by Jonas Karlman (Kwiboo) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20847 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20847.patch This is a follow up to a very old series from [April 2019 (RFC)](https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2019-April/242316.html), [December 2020 (v1)](https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2020-December/273579.html) and [August 2024 (v2)](https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2024-August/332034.html), adding V4L2 Request API hwaccels for stateless decoding of MPEG2, H.264, HEVC and VP8. These hwaccels has in one form or another been used in [LibreELEC](https://github.com/LibreELEC/LibreELEC.tv) community, nightly and release images since Dec 20, 2018. A version very close to the one submitted in this PR has been used by LibreELEC since Dec 22, 2024. See [August 2024 (v2)](https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2024-August/332034.html) for a short summary of the history behind this PR. **CHANGES** In this version (and v2) a new V4L2 Request API hwdevice has been added. Thanks to this it is now possible to specify what media device to use for decoding, in case multiple decoders exist on a system. E.g. using a `-init_hw_device v4l2request:/dev/media1` parameter. Compared to v2 this only includes support for `V4L2_PIX_FMT`s that has been merged into mainline Linux. As a result, this version cannot be tested with the downstream `rpivid` driver on Raspberry Pi or using AFBC buffers on Allwinner. **HOW TO USE** To use the V4L2 Request API hwaccels you must build FFmpeg on a system with recent Linux kernel headers, `v6.0+`. It also requires `libdrm` and `libudev` for the hwaccels to be enabled and successfully build. This can then be runtime tested on multiple Allwinner and Rockchip devices. ``` ffmpeg -hwaccel v4l2request -hwaccel_output_format drm_prime -i <input-path> -map 0:v -f null - ``` This series has been tested with `cedrus` driver on Allwinner H6, `hantro` and `rkvdec` driver on Rockchip RK3399. It should be possible to playback video using `kodi-gbm` and `mpv` (with [PR14690](mpv-player/mpv#14690)) using the v4l2request hwaccels on supported devices. It should also be possible to run [fluster](https://github.com/fluendo/fluster) test suites with [PR179](fluendo/fluster#179). **FUTURE** Following parts has been held back and will follow in future PRs. - Support `V4L2_PIX_FMT`s used by pending [rpi-hevc-dec](https://lore.kernel.org/linux-media/20250701-media-rpi-hevc-dec-v4-0-057cfa541177@raspberrypi.com/) driver. - Support for VP9 and AV1 codecs. - Check for PROFILE and LEVEL controls to filter out any video device not reporting support for current profile and level. - Estimate max buffer size instead of using a fixed value. >From 0e4ba193b437de4fd011308ea0a3fa7176dd16ee Mon Sep 17 00:00:00 2001 From: Jonas Karlman <jonas@kwiboo.se> Date: Tue, 6 Aug 2024 09:06:00 +0000 Subject: [PATCH 1/9] avutil/hwcontext: Add hwdevice type for V4L2 Request API Add a hwdevice type for V4L2 Request API with transfer_data_from support for AV_PIX_FMT_DRM_PRIME, based on AV_HWDEVICE_TYPE_DRM. AVV4L2RequestDeviceContext.media_fd can be set by the application or a media device path can be supplied when hwdevice is created. When none is supplied it default to -1 and hwaccel will auto-detect a media device with a capable video device. Signed-off-by: Jonas Karlman <jonas@kwiboo.se>
Kwiboo
force-pushed
the
ffmpeg-v4l2request
branch
3 times, most recently
from
9018f32 to
53afa11
Compare
Add support for FFmpeg V4L2 Request API hwaccel decoders. Signed-off-by: Jonas Karlman <jonas@kwiboo.se>
mdimopoulos
force-pushed
the
ffmpeg-v4l2request
branch
from
53afa11 to
22ca8f5
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR add initial support for FFmpeg V4L2 Request API hwaccel decoders.
Created as a draft PR because work-in-progress FFmpeg patches are still being worked on and changes may be required after review.
This also uses a hwdownload mapping that may not work with all V4L2 Request API drivers, this has mainly been tested using
cedrus,hantroandrkvdecdrivers and they mostly use thenv12format.Possible solutions to the hwdownload format issue could be to:
Will keep this PR in draft until FFmpeg patches have been accepted by upstream.