[FFmpeg-devel] [PATCH 2/2] nut: per frame side and meta data support

[Reimar Döffinger]

On 23.12.2013, at 03:21, Michael Niedermayer <michaelni at gmx.at> wrote:
> This variant is simpler and has less overhead than the previous
> It is not compatible with existing demuxers though, the reasoning here
> is that as side data is essential for presentation of a stream demuxer support
> is neccessary anyway.

This seems to split things into "side data" which is defined as essential for decoding and metadata.
But this seems a bit too limited to me. For example I believe we now have side data in FFmpeg that contains CC subtitle data. It is clearly not necessary for decoding and it clearly isn't metadata either.
I guess it maybe shouldn't be written in this structure at all, but would/will the FFmpeg muxer do something sensible with it?
I think it would be good to make sure we have enough flexibility to represent stuff we did not yet think of, though I fully approve of the "essential to decode" distinction in principle. It's just that we don't really have this information if its essential in FFmpeg, so I'd suspect we might end up generating lots of invalid files (due to them having non-essential data in side data for example).