[FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg

[Philip Langdale]

On Wed, 08 Feb 2023 19:38:07 +0200
Rémi Denis-Courmont <remi at remlab.net> wrote:

> Le keskiviikkona 8. helmikuuta 2023, 16.27.52 EET Devin Heitmueller a
> écrit :
> > On Wed, Feb 8, 2023 at 8:08 AM Jean-Baptiste Kempf
> > <jb at videolan.org> wrote:  
> > > Do we really need those PIX_FMT in FFmpeg public API?  
> > 
> > Tiled formats are annoying, but not uncommon in popular embedded
> > platforms (e.g. NXP iMX, TI Davinci).   
> 
> Uh, not to deny that, but tiled formats are in no way an embedded
> thing, as you imply. They are also used by desktop GPUs. And that is
> not really relevant, I would argue, to the discussion, either way.
> 
> > Of course ffmpeg could choose
> > to ignore them, but it would effectively prevent it from being used
> > on those platforms (and it's pretty essential to use the hardware
> > blocks to do any real video processing).  
> 
> Insofar as the format is *only* used by a single module, I don't see
> the need to assign it a pixel format in libavutil. You could just as
> well define a generic V4L format that would be used for all weird V4L
> formats that nothing other than V4L code understands.
> 
> Ditto tiled DRM pixel formats for that matter. And while V4L is maybe
> not there yet, DRM has introduced format modifiers that anyway
> preclude any simple enumeration. My point being, there needs to be a
> way to further specify framework-specific formats outside of the
> avutil pixel format list, in any case.
> 

I never thought about it this way until you mentioned DRM but I think
that's the key insight.

If we want to take a leaf out of the DRM book (and wouldn't it be
wonderful if these m2m decoders used DRM formats?), then this format
would NV12 at the pix fmt layer with a modifier indicating the tiling
pattern. When we interact with these kinds of formats in VAAPI, for
example, VAAPI encapsulates the heavy lifting required to transfer
to/from system memory successfully, and when we are doing everything on
the GPU side, we pass the surface long with the modifier to whatever
eventually consumes it (eg: Display with GL or Vulkan, or encoding). In
none of these cases does ffmpeg code really need to care about the
modifier. Maybe we've been thinking about these m2m formats the wrong
way.

--phil