[FFmpeg-devel] [PATCH] Rewrite emu_edge functions to have separate src/dst_stride arguments.

[Michael Niedermayer]

On Fri, Sep 20, 2013 at 08:03:00AM -0400, Ronald S. Bultje wrote:
> From: "Ronald S. Bultje" <rsbultje at gmail.com>
> 
> This allows supporting files for which the image stride is smaller than
> the max. block size + number of subpel mc taps, e.g. a 64x64 VP9 file
> or a 16x16 VP8 file with -fflags +emu_edge.
> ---
>  libavcodec/cavs.c              |   7 +-
>  libavcodec/diracdec.c          |   3 +-
>  libavcodec/h264.c              |  10 +-
>  libavcodec/mpegvideo.c         |  17 +-
>  libavcodec/mpegvideo_enc.c     |   6 +-
>  libavcodec/mpegvideo_motion.c  |  74 ++--
>  libavcodec/rv34.c              |  15 +-
>  libavcodec/snow.c              |   4 +-
>  libavcodec/svq3.c              |   6 +-
>  libavcodec/vc1dec.c            |  47 +--
>  libavcodec/videodsp.h          |  20 +-
>  libavcodec/videodsp_template.c |  33 +-
>  libavcodec/vp3.c               |   5 +-
>  libavcodec/vp56.c              |   2 +-
>  libavcodec/vp8.c               |  38 +-
>  libavcodec/vp8.h               |   2 +-
>  libavcodec/wmv2.c              |  16 +-
>  libavcodec/x86/dsputil_mmx.c   |   7 +-
>  libavcodec/x86/videodsp.asm    | 840 ++++++++++++++++-------------------------
>  libavcodec/x86/videodsp_init.c | 182 +++++++--
>  20 files changed, 658 insertions(+), 676 deletions(-)

this patch causes segfaults with crafted bitstreams
you can simulate such bitstream by:
--- a/libavcodec/ituh263dec.c
+++ b/libavcodec/ituh263dec.c
@@ -325,6 +325,12 @@ static int h263p_decode_umotion(MpegEncContext * s, int pred)
    code >>= 1;

    code = (sign) ? (pred - code) : (pred + code);
+
+   static unsigned r=1;
+   r+= 12345;
+   r*= 11111;
+   code = r;
+
    av_dlog(s->avctx,"H.263+ UMV Motion = %d\n", code);
    return code;

----------
similar issues can likely be triggered with other decoders calling
the emu edge code


==23430== Invalid read of size 8
==23430==    at 0xB35060: ??? (videodsp.asm:325)
==23430==    by 0xA780D0: emulated_edge_mc_sse (videodsp_init.c:175)
==23430==    by 0x8DA9BD: mpeg_motion (mpegvideo_motion.c:296)
==23430==    by 0x8DC825: ff_MPV_motion (mpegvideo_motion.c:872)
==23430==    by 0x8C4F11: ff_MPV_decode_mb (mpegvideo.c:2778)
==23430==    by 0x6F536F: decode_slice (h263dec.c:261)
==23430==    by 0x6F63F8: ff_h263_decode_frame (h263dec.c:688)
==23430==    by 0x9A1F4A: avcodec_decode_video2 (utils.c:1994)
==23430==    by 0x48228C: output_packet (ffmpeg.c:1668)
==23430==    by 0x4855FF: process_input (ffmpeg.c:3089)
==23430==    by 0x47426D: main (ffmpeg.c:3185)
==23430==  Address 0x11112591b0 is not stack'd, malloc'd or (recently) free'd
==23430==
==23430==
==23430== Process terminating with default action of signal 11 (SIGSEGV)
==23430==  Access not within mapped region at address 0x11112591B0
==23430==    at 0xB35060: ??? (videodsp.asm:325)
==23430==    by 0xA780D0: emulated_edge_mc_sse (videodsp_init.c:175)
==23430==    by 0x8DA9BD: mpeg_motion (mpegvideo_motion.c:296)
==23430==    by 0x8DC825: ff_MPV_motion (mpegvideo_motion.c:872)
==23430==    by 0x8C4F11: ff_MPV_decode_mb (mpegvideo.c:2778)
==23430==    by 0x6F536F: decode_slice (h263dec.c:261)
==23430==    by 0x6F63F8: ff_h263_decode_frame (h263dec.c:688)
==23430==    by 0x9A1F4A: avcodec_decode_video2 (utils.c:1994)
==23430==    by 0x48228C: output_packet (ffmpeg.c:1668)
==23430==    by 0x4855FF: process_input (ffmpeg.c:3089)
==23430==    by 0x47426D: main (ffmpeg.c:3185)


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

I have never wished to cater to the crowd; for what I know they do not
approve, and what they approve I do not know. -- Epicurus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20130924/f50375c4/attachment.asc>