[FFmpeg-devel] h264_annexbtomp4 filter ideas, was: Collection of patches

[Thorsten Jordan]

Thorsten Jordan schrieb:
> Michael Niedermayer schrieb:
>> On Wed, Apr 23, 2008 at 09:15:40AM +0200, Thorsten Jordan wrote:
>> [...]
> 
>>> work. Maybe it has been fixed differently meanwhile, but i can't
>>> imaginge how.
>> Your patch generates invalid/broken h264. The current code generates broken
>> ones too but they dont work which prevents them from spreading. Thus your
>> patch clearly makes it worse, the issue is not fixed it just appears that it
>> is. The spliting API in ffmpeg has to be changed to allow spliting random
>> subsets of the headers out. Please see the mailing list where i described
>> this more precissely IIRC.
> ah now i see your point. The splitter should thus extract the SPS and
> PPS only and put it in extradata, and leave the rest of the stream as it
> is (including the SEI NALs). If there are NALs other than SPS/PPS
> inside, it would be incompatible to MP4 extradata format, as it expects
> only SPS and PPS units.
> So if there is "SPS - SEI - PPS - other/rest..." in the stream, extract
> "SPS - PPS" to extra data, and keep "SEI - other/rest..." in the stream
> 
> In the discussion about that problem on this list, you said that the two
> SEI NALs in my example shouldnt be part of global headers. I checked the
> h264 spec and there are 22 types of SEIs. None of them sound like they
> are SPS/PPS related.
> 
> Which brings up the question if there should be never SEI units in
> global headers. I found nothing near the SPS/PPS description that
> mentions SEI, so they are not used as header extension?

If one would solve this by writing a h264_annexbtomp4 bitstream filter,
there are some issues:

The global headers (extradata) of MP4 should contain SPS and PPS only,
but it can contain multiple SPS/PPS afaics.

This means the bitstream filter has to scan the whole h264 elementary
stream, rip out every SPS+PPS inside, and collect them in one global
block. For every SPS/PPS in the stream it must compare it to the last
found ones to avoid storing the same SPS/PPS twice. I suppose there is a
limit on the number of SPS/PPS, as they need to be given to the decoder
all at once on decoding - but if there are more than one SPS/PPS each
the reverse filter mp4toannexb wouldn't work...

This means the extradata (global headers) are not ready before the full
stream has been filtered. I don't know if that is a problem for the mp4
muxer.

Where should the collected headers get stored? Into the
avcodeccontext.extradata pointer of the given avcodeccontext?

Should such a filter change the Stream-Mode (with startcodes) to MP4
package mode also (would be incompatible to current mp4 muxer i guess)?

-- 
Regards, Thorsten