[FFmpeg-devel] FFMPEG for V4L2 M2M devices ?

[Dave Stevenson]

On Mon, 12 Jul 2021 at 14:51, Andrii <andrii.liubimov at gmail.com> wrote:
>>
>> A quick Google implies that NVidia already has a stateful V4L2 M2M
>> driver in their vendor kernel. Other than the strange choice of device
>> node name (/dev/nvhost-nvdec), the details at [3] make it look like a
>> normal V4L2 M2M decoder that has a good chance of working against
>> h264_v4l2m2m.
>
>
> Not only does it have a strange node name, it also uses two nodes. One for decoding, another for converting. Capture plane of the decoder stores frames in V4L2_PIX_FMT_NV12M format.
> Converter able to convert it to a different format[1].

Those appear to be two different hardware blocks.
If you can consume NV12M (YUV420 with interleaved UV plane), then I
see no reason why you have to pass the data through the
"/dev/nvhost-vic" device.

We have a similar thing where /dev/video10 is the decoder (stateful
decode), /dev/video11 is the encoder, and /dev/video12 is the ISP
(Image Sensor Pipeline) wrapped in the V4L2 API.

> Could you point me at documentation of Pi V4L2 spec?

It just implements the relevant APIs that I've already linked to. If
it doesn't follow the API, then we fix it so that it does.

Stateful H264 implementation is
https://github.com/raspberrypi/linux/tree/rpi-5.10.y/drivers/staging/vc04_services/bcm2835-codec
Stateless HEVC is
https://github.com/raspberrypi/linux/tree/rpi-5.10.y/drivers/staging/media/rpivid

  Dave

> [1] https://docs.nvidia.com/jetson/l4t-multimedia/group__V4L2Conv.html
>
> Andrii
>
> On Mon, Jul 12, 2021 at 6:02 AM Dave Stevenson <dave.stevenson at raspberrypi.com> wrote:
>>
>> On Sat, 10 Jul 2021 at 00:56, Brad Hards <bradh at frogmouth.net> wrote:
>> >
>> > On Saturday, 10 July 2021 8:53:27 AM AEST Andrii wrote:
>> > > I am working on porting a Kodi player to an NVidia Jetson Nano device. I've
>> > > been developing a decoder for quite some time now, and realized that the
>> > > best approach would be to have it inside of ffmpeg, instead of embedding
>> > > the decoder into Kodi as it heavily relies on FFMPEG. Just wondering if
>> > > there is any effort in making FFMPEG suppring M2M V4L devices ?
>> >
>> > https://git.ffmpeg.org/gitweb/ffmpeg.git/blob_plain/HEAD:/libavcodec/v4l2_m2m.c[1]
>> >
>> > I guess that would be the basis for further work as required to meet your needs.
>>
>> Do note that there are 2 V4L2 M2M decoder APIs - the stateful API[1] ,
>> and the stateless API [2]. They differ in the amount of bitstream
>> parsing and buffer management that the driver implements vs expecting
>> the client to do.
>>
>> The *_v4l2m2m drivers within FFMPEG support the stateful API (ie the
>> kernel driver has bitstream parsing). For Raspberry Pi we use that to
>> support the (older) H264 implementation, and FFMPEG master does that
>> very well.
>>
>> The Pi HEVC decoder uses the V4L2 stateless API. Stateless HEVC
>> support hasn't been merged to the mainline kernel as yet, so there are
>> downstream patches to support that.
>>
>> A quick Google implies that NVidia already has a stateful V4L2 M2M
>> driver in their vendor kernel. Other than the strange choice of device
>> node name (/dev/nvhost-nvdec), the details at [3] make it look like a
>> normal V4L2 M2M decoder that has a good chance of working against
>> h264_v4l2m2m.
>>
>> [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-decoder.html
>> [2] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-stateless-decoder.html
>> [3] https://docs.nvidia.com/jetson/l4t-multimedia/group__V4L2Dec.html
>>
>>   Dave
>>
>> > Brad
>> >
>> > --------
>> > [1] https://git.ffmpeg.org/gitweb/ffmpeg.git/blob_plain/HEAD:/libavcodec/v4l2_m2m.c
>> > _______________________________________________
>> > ffmpeg-devel mailing list
>> > ffmpeg-devel at ffmpeg.org
>> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> >
>> > To unsubscribe, visit link above, or email
>> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".