[Ffmpeg-devel] h.264 decoder segfault

[Michael Niedermayer]

Hi

On Tue, Jan 23, 2007 at 05:58:27PM +0100, Benoit Fouet wrote:
> Hi,
> 
> Michael Niedermayer wrote:
> > does it also crash under gdb?
> yes, it does
> 
> >  if so maybe a backtrace from gdb would be
> > more informative ...
> >   
> what about this backtrace ?
> 
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread -1214749008 (LWP 24106)]
> 0x08235895 in decode_residual (h=0xb788f020, gb=0xb7891100,
> block=0xb78c8da8, n=0,
>     scantable=0xb78c9635
> "\001\004\b\005\002\003\006\t\f\r\n\a\v\016\017", qmul=0xb78aac24,
> max_coeff=15) at bitstream.h:888
> 888         GET_VLC(code, re, s, table, bits, max_depth)
> (gdb) bt
> #0  0x08235895 in decode_residual (h=0xb788f020, gb=0xb7891100,
> block=0xb78c8da8, n=0,
>     scantable=0xb78c9635
> "\001\004\b\005\002\003\006\t\f\r\n\a\v\016\017", qmul=0xb78aac24,
> max_coeff=15) at bitstream.h:888
> #1  0x0824a82d in decode_mb_cavlc (h=0xb788f020) at h264.c:5487
> #2  0x0825cecc in decode_slice (h=0xb788f020) at h264.c:7435
> #3  0x0825e112 in decode_nal_units (h=0xb788f020, buf=0x85d1b70 "",
> buf_size=4509) at h264.c:8146
> #4  0x0825fb93 in decode_frame (avctx=0x84760c0, data=0xbfa2dbc0,
> data_size=0xbfa2d830, buf=0x85d1b70 "", buf_size=4509) at h264.c:8316
> #5  0x080c529e in avcodec_decode_video (avctx=0x84760c0, picture=0x0,
> got_picture_ptr=0xbfa2d830, buf=0x0, buf_size=140319600)
>     at utils.c:904
> #6  0x0805b6a9 in output_packet (ist=0x84770d0, ist_index=0,
> ost_table=0x8477120, nb_ostreams=1, pkt=0xbfa2de80) at ffmpeg.c:1092
> #7  0x080635f6 in main (argc=-1079845248, argv=0x0) at ffmpeg.c:1936
> 
> Ben
> 
> PS: gdb is still running, if you need anything else :)

the linenumber in decode_residual() :)))
which is definitly not in bitstream.h but h264.c
i do know its crashing in get_vlc*() which is called by decode_residual()
but i dont know in which, there are several
one of the vlc tables either is damaged, or more likely a index into one
is out of range

maybe a "#define always_inline" before #include "bitstream.h" with -O0
would help

or the good old add a printf() before each get_vlc*() in decode_residual()

PS: fuzzer crash/bugreports about other decoders and demuxers are also very
welcome

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

No evil is honorable: but death is honorable; therefore death is not evil.
-- Citium Zeno
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20070123/5278f897/attachment.pgp>