[FFmpeg-devel] [PATCH 000/279 v2] New channel layout API

[Michael Niedermayer]

On Sun, Dec 19, 2021 at 12:35:11PM +0100, Marton Balint wrote:
> 
> 
> On Sat, 18 Dec 2021, Michael Niedermayer wrote:
> 
> > On Sat, Dec 18, 2021 at 02:36:12PM +0100, Michael Niedermayer wrote:
> > > On Fri, Dec 17, 2021 at 07:04:08PM +0100, Marton Balint wrote:
> > > > 
> > > > 
> > > > On Fri, 17 Dec 2021, Michael Niedermayer wrote:
> > > > 
> > > > > On Fri, Dec 17, 2021 at 01:04:19AM +0100, Marton Balint wrote:
> > > > > > 
> > > > > > 
> > > > > > On Thu, 16 Dec 2021, James Almer wrote:
> > > > > > 
> > > > > > > Resending the first two patches only, since this is meant to
> > > > > > > show the implementation of one of the several suggestions made
> > > > > > > in the previous set that need to be discussed and hopefully
> > > > > > > resolved in a call.
> > > > > > 
> > > > > > Can you push the full branch somewhere?
> > > > > > 
> > > > > > > 
> > > > > > > The proposals so far to extend the API to support either custom
> > > > > > > labels for channels are, or some form of extra user information.
> > > > > > > 
> > > > > > > - Fixed array of bytes to hold a label. Simple solution, but
> > > > > > >  the labels will have a hard limit that can only be extended
> > > > > > >  with a major bump. This is what i implemented in this version.
> > > > > > > - "char *name" per channel that the user may allocate and the
> > > > > > >  API will manage, duplicate and free. Simple solution, and the
> > > > > > >  name can be arbitrarily long, but inefficient (av_strdup() per
> > > > > > >  channel with a custom label on layout copy).
> > > > > > > - "const char *name" per channel for compile time constants, or
> > > > > > >  that the user may allocate and free. Very efficient, but for
> > > > > > >  non compile time strings ensuring they outlive the layout can
> > > > > > >  be tricky.
> > > > > > > - Refcounted AVChannelCustom with a dictionary. This can't be
> > > > > > >  done with AVBufferRef, so it would require some other form
> > > > > > >  of reference counting. And a dictionary may add quite a bit of
> > > > > > >  complexity to the API, as you can set anything on them.
> > > > > > 
> > > > > > Until we have proper refcounting API we can make the AVBufferRef in
> > > > > > AVChannelLayout a void *, and only allow channel_layout functions to
> > > > > > dereference it as an AVBufferRef. This would mean adding some extra helper
> > > > > > functions to channel layout, but overall it is not unsolvable.
> > > > > > 
> > > > > > The real question is that if you want to use refcounting and add helpers to
> > > > > > query / replace per-channel metadata, or you find the idea too heavy weight
> > > > > > and would like to stick to flat structs.
> > > > > 
> > > > > what is the advantage of refcounting for channel metadata ?
> > > > > is it about the used memory, about the reduced need to copy ?
> > > > 
> > > > Basicly it is the ability to store per-channel metadata in avdictionary,
> > > > because otherwise it would have to be copyed, and avdictionary is very
> > > > ineffective at copying because of many mallocs.
> > > > 
> > > > > 
> > > > > what kind of metadata and what size do you expect ?
> > > > > bytes, kilobytes, megabytes, gigabytes per channel ?
> > > > 
> > > > Usually, nothing, because most format don't have support for per-channel
> > > > metadata. In some cases it is going to be a couple of textual metadata
> > > > key-value pairs, such as language, label, group, speaker, positon, so 4-5
> > > > dynamically allocated string pairs, plus the AVDictionary itself, multiplied
> > > > by the number of channels in a layout.
> > > > 
> > > > > 
> > > > > what is the overhead for dynamic allocation and ref counting?
> > > > > that is at which point does it even make sense ?
> > > > 
> > > > I don't have exact measurements. It is generally felt that copying
> > > > AVDictionary per-channel is a huge overhead for something as lightweight as
> > > > an audio frame which is a 2-4 kB per channel at most and only a couple of
> > > > allocs usually not dependant on the number of channels. That's why
> > > > refcounting was proposed.
> > > 
> > > I was thinking more at a AVStream / AVCodecParameters level.
> > 
> > > How will a demuxer transport such metadata over a AVPacket into a decoder
> > > outputting metadata-filled AVFrames?
> > 
> > or is this never needed ?
> 
> I am not sure I understand. Usually metadata is passed from demuxer to
> decoder by avcodec_parameters_to_context(), this is used for all metadata
> which is in AVCodecParameters.
> 

> For per-packet metadata ff_decode_frame_props() has some automatic packet
> side data -> frame side data transfer.
> 
> AVStream side data may be transferred to AVPacket side data if
> av_format_inject_global_side_data() is used, but it is not enabled by
> default.

The sidedata in AVPacket is not channel specific, the data in AVFrames new
channels is.
The later is as you wrote expensive to copy/alloc so it needs ref counting

what i was trying to point to was that if we need a way to transfer this
data on a per packet base from demuxer forward then we need a flat format.
At least with the current AVPackets
And a flat format is much lighter to copy around so then one is pushed
toward the question, "should that be used for AVFrames too?"
a small <10kb audio frame with 8 audio channels and a ref counted dictionary
or whatever per each of the 8 channels vs. a flat description.
an extra is that a flat format is very simple to memcmp() to check if it
changed.
I have no position in this above, iam just wanting to make sure nothing
is missed in this somewhat rushed design.

also theres yet another orthogonal aspect which has been missed i think
This per channel description whatever its form will be is usefull for another
case. And that are the video/image channels. generally this is 
red/green/blue/(alpha) but there is material with other channels, 
IR/NIR/UV/... and i imagine probably depth from some lidar/radar system

Ideally such a future image channel API can share parts with the audio one

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Opposition brings concord. Out of discord comes the fairest harmony.
-- Heraclitus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20211219/0f5679a8/attachment.sig>