[FFmpeg-devel] [PATCH 3/4 v2] avcodec/vc1: rewrite vc1_decode_i_blocks to align with VC-1 spec

[Jerome Borsboom]

Thank you for the rigorous testing of my patches. I try to be careful
when changing things, but every now and then I unwittingly may break
things that do not surface in setup.

There is something strange with the backtraces. The 'Slice header
damaged' error indicates that the source file is coded with slices.
Slices, however, only occur in VC-1 advanced profile. The backtrace with
the invalid read contains vc1_decode_i_blocks which is only used in
Simple and Main profile. I currently do not see how this adds up.

Is this crash related to a specific source file? If so, could you share it?


Regards,
Jerome

> 
> crashes:
> 
> [vc1 @ 0x11b77200] Slice header damaged
> ==2065==    at 0x120D69C: VALGRIND_PRINTF_BACKTRACE (valgrind.h:4550)
> ==2065==    by 0x120E11C: av_log_default_callback (log.c:351)
> ==2065==    by 0x120E2BB: av_vlog (log.c:377)
> ==2065==    by 0x120E27B: av_log (log.c:369)
> ==2065==    by 0xC3B5B5: vc1_decode_frame (vc1dec.c:1014)
> ==2065==    by 0x8A6BAB: decode_simple_internal (decode.c:398)
> ==2065==    by 0x8A7832: decode_simple_receive_frame (decode.c:594)
> ==2065==    by 0x8A78FD: decode_receive_frame_internal (decode.c:612)
> ==2065==    by 0x8A7B75: avcodec_send_packet (decode.c:674)
> ==2065==    by 0x43373E: decode (ffmpeg.c:2238)
> ==2065==    by 0x433F98: decode_video (ffmpeg.c:2382)
> ==2065==    by 0x434FE0: process_input_packet (ffmpeg.c:2623)
> ==2065==    by 0x43C207: process_input (ffmpeg.c:4461)
> ==2065==    by 0x43C7B0: transcode_step (ffmpeg.c:4581)
> ==2065==    by 0x43C92C: transcode (ffmpeg.c:4635)
> ==2065==    by 0x43D199: main (ffmpeg.c:4842)
> ==2065== Invalid read of size 8
> ==2065==    at 0xD790BD: ??? (libavcodec/x86/vc1dsp_loopfilter.asm:302)
> ==2065==    by 0xC26AA6: vc1_i_v_loop_filter (vc1_loopfilter.c:239)
> ==2065==    by 0xC26BFB: ff_vc1_i_loop_filter (vc1_loopfilter.c:266)
> ==2065==    by 0xC23C72: vc1_decode_i_blocks (vc1_block.c:2594)
> ==2065==    by 0xC259D4: ff_vc1_decode_blocks (vc1_block.c:2969)
> ==2065==    by 0xC3B7D6: vc1_decode_frame (vc1dec.c:1042)
> ==2065==    by 0x8A6BAB: decode_simple_internal (decode.c:398)
> ==2065==    by 0x8A7832: decode_simple_receive_frame (decode.c:594)
> ==2065==    by 0x8A78FD: decode_receive_frame_internal (decode.c:612)
> ==2065==    by 0x8A7B75: avcodec_send_packet (decode.c:674)
> ==2065==    by 0x43373E: decode (ffmpeg.c:2238)
> ==2065==    by 0x433F98: decode_video (ffmpeg.c:2382)
> ==2065==    by 0x434FE0: process_input_packet (ffmpeg.c:2623)
> ==2065==    by 0x43C207: process_input (ffmpeg.c:4461)
> ==2065==    by 0x43C7B0: transcode_step (ffmpeg.c:4581)
> ==2065==    by 0x43C92C: transcode (ffmpeg.c:4635)
> ==2065==    by 0x43D199: main (ffmpeg.c:4842)
> ==2065==  Address 0x11f25100 is not stack'd, malloc'd or (recently) free'd
> 
> ....
> 
> ==2065== Process terminating with default action of signal 11 (SIGSEGV)
> ==2065==  General Protection Fault
> ==2065==    at 0x13786A6: ??? (in ffmpeg/ffmpeg_g)
> ==2065==    by 0x11F8D78: av_buffer_pool_uninit (buffer.c:285)
> ==2065==    by 0xBF5F7B: avcodec_close (utils.c:1089)
> ==2065==    by 0x43CCB5: transcode (ffmpeg.c:4697)
> ==2065==    by 0x43D199: main (ffmpeg.c:4842)
> 
> [...]