[Ffmpeg-devel] assertion failure in epzs_motion_search_internal

[Trent Piepho]

I'm getting an assertion failure:

mencoder: motion_est_template.c:1044: epzs_motion_search_internal: Assertion `(P[3][0]>>shift) <= xmax' failed.

This is when encoding some 1280x720 HD material with nearly latest svn,
r7245.

Options used with mencoder are:

vcodec=mpeg4:qpel:v4mv:psnr:qprd:vbitrate=6950:vqmin=2:lmin=1:dc=8:trell:
mbd=2:precmp=2:subcmp=2:cmp=2:dia=-4:predia=8:cbp:mv0:sc_factor=4:
vmax_b_frames=2:vb_strategy=2:b_sensitivity=20:brd_scale=1:bidir_refine=1:autoaspect:vpass=1

I'm trying to find a sample that will trigger the error without 5 or 6
hours of encoding first.

I had this problem before and was able to trace it up farther.

ff_estimate_motion_b() was setting P_TOPRIGHT[0] or P_TOP[0] to values that
were out of range.  Around line 1517 in motion_est.c.  The values were not
copletely invalid, but just invalid for that macroblock.  For example,
something like 1200 for the P_TOP[0], which would be ok for some
macroblocks in a 1280 pixel side source, just not for the one it
was getting assigned too.

I also noticed that mot_stride, the stride of mv_table, was 81.  1280/16 =
80.  Seemed odd to me, but maybe there's supposed to be an extra block.