[FFmpeg-devel] [RFC]x264 avcc and related issue

[Zhao Zhili]

There are multiple encoders which can output bitstream in avcc/hvcc format,
e.g, x264, x265 and videotoolbox. This can improve performance depends on
container format (no more ff_avc_parse_nal_units_buf).

The problem is these encoders output parameter sets directly, not contained
in a complete AVC/HEVC configuration box. But muxers only expect 
extradata in
annexb format, or a complete AVC/HEVC configuration box, not support such
length+sps+length+pps format. The following command will output a broken 
file:

./ffmpeg -i input.mp4 -an -c:v libx264 -x264-params annexb=0 broken.mp4

I want to fix this issue and improve the performance at the same time.

There are multiple options that I don't know how to make a choice.

1. At the muxer level? It's easy and ugly. And there are corner case
like 00 00 01 xx can be start code of annexb and size of avcc at the
same time.

2. A new bsf? It looks less useful outside of libavcodec. And encoder
can't enable a bsf automatically.

3. A specific helper like h26x_vcc.h? Can it be more general?

Any more suggestions?