[Ffmpeg-devel] h.264 decoder segfault

[Benoit Fouet]

Hi,

Michael Niedermayer wrote:
> the linenumber in decode_residual() :)))
>   
granted :)

> which is definitly not in bitstream.h but h264.c
> i do know its crashing in get_vlc*() which is called by decode_residual()
> but i dont know in which, there are several
> one of the vlc tables either is damaged, or more likely a index into one
> is out of range
>
> maybe a "#define always_inline" before #include "bitstream.h" with -O0
> would help
>
> or the good old add a printf() before each get_vlc*() in decode_residual()
>   
here is the one that causes the SEGFAULT:
first line is line 4957
if(total_coeff == max_coeff)
        zeros_left=0;
    else{
        if(n == CHROMA_DC_BLOCK_INDEX)
            zeros_left= get_vlc2(gb, chroma_dc_total_zeros_vlc[
total_coeff-1 ].table, CHROMA_DC_TOTAL_ZEROS_VLC_BITS, 1);
        else
            zeros_left= get_vlc2(gb, total_zeros_vlc[ total_coeff-1
].table, TOTAL_ZEROS_VLC_BITS, 1);
    }


the second one causes the crash...
when it crashes, gb is correct (same as previous calls)
total_zeros_vlc[ total_coeff-1 ].table = NULL

Hope that helps more...

> PS: fuzzer crash/bugreports about other decoders and demuxers are also very
> welcome
>   
if i find other ones, i'll send them here :)

Ben