[FFmpeg-devel] [NUT-devel] [PATCH 2/2] nut: per frame side and meta data support

[Michael Niedermayer]

On Mon, Dec 23, 2013 at 01:28:16PM +0100, Reimar Döffinger wrote:
> On 23.12.2013, at 03:21, Michael Niedermayer <michaelni at gmx.at> wrote:
> > This variant is simpler and has less overhead than the previous
> > It is not compatible with existing demuxers though, the reasoning here
> > is that as side data is essential for presentation of a stream demuxer support
> > is neccessary anyway.
> 
> This seems to split things into "side data" which is defined as essential for decoding and metadata.
> But this seems a bit too limited to me. For example I believe we now have side data in FFmpeg that contains CC subtitle data.
> It is clearly not necessary for decoding and it clearly isn't metadata either.

its not neccessary for decoding the video but it is neccessary
for presentation of the video where audio cannot be used, like in case
of a loud environment, hearing impairment or insufficient language
compatibility between the language used in the audio and the listener
so i would argue that it qualifies as side data if its not in its
own stream/track


> I guess it maybe shouldn't be written in this structure at all, but would/will the FFmpeg muxer do something sensible with it?

Iam not sure which side data type you speak of exactly?


> I think it would be good to make sure we have enough flexibility to represent stuff we did not yet think of,

Its hard to make sure that we support things that we did not think
of.
If one asks the question "what did we not think of"
well maybe side data per slice or per macroblock
or metadata for a subset of pixels in a picture
These could be supprted by including the "where/what" in the "key/name"
tough like
"Person-RectXY[50,70-200,30]", "guy fawkes"
"Slice56-FirstMV", "12.25,89"
iam not sure how usefull that is though ...



> though I fully approve of the "essential to decode" distinction in principle.
> It's just that we don't really have this information if its essential in FFmpeg,
> so I'd suspect we might end up generating lots of invalid files (due to them
> having non-essential data in side data for example).

id assume all metadata would use AV_PKT_DATA_STRINGS_METADATA or
AV_PKT_DATA_METADATA_UPDATE and the rest would be side data.
if a new field that represents metadata is added to ffmpeg but not
to nut and if our nut muxer would store unknown types (it does not
currently, but would with the proposed patch, but that could be
changed)
and the input would contain that new field then the muxer would create
a file that contains a frame with a side data element that is "Junk"
a demuxer either would ignore the side data or extract it back into
the new metadata field opaquely.
While this would be incorrect in some sense, as it should have been
stored in metadata instead, i dont see how it could break anything.
Also there are quite a few ifs in there and id consider such "storing
in the wrong list" a bug in the implementation

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

It is what and why we do it that matters, not just one of them.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20131223/891efb94/attachment.asc>