[Ffmpeg-devel] assertion failure in epzs_motion_search_internal

[Trent Piepho]

On Mon, 11 Dec 2006, Michael Niedermayer wrote:
> On Thu, Dec 07, 2006 at 05:01:52PM -0800, Trent Piepho wrote:
> > mencoder: motion_est_template.c:1044: epzs_motion_search_internal: Assertion `(P[3][0]>>shift) <= xmax' failed.
> >
> > This is when encoding some 1280x720 HD material with nearly latest svn,
> > r7245.
> >
> > Options used with mencoder are:
> >
> > vcodec=mpeg4:qpel:v4mv:psnr:qprd:vbitrate=6950:vqmin=2:lmin=1:dc=8:trell:
> > mbd=2:precmp=2:subcmp=2:cmp=2:dia=-4:predia=8:cbp:mv0:sc_factor=4:
> > vmax_b_frames=2:vb_strategy=2:b_sensitivity=20:brd_scale=1:bidir_refine=1:autoaspect:vpass=1
> >
>
> if "MV.x" is too large for the current macroblock then so it is for
> the macroblock above and aboveright (P_TOP and P_TOPRIGHT)
> so it _seems_ that an illegal motion vector has already been used in a
> previous macroblock ...

Now that I'm able to repoduce this after only 10 minutes or so of encoding,
I've tracked it back farther.

ff_estimate_b_frame_motion: estimate b forward
ff_estimate_motion_b: 72,18 P_TOPRIGHT is <4092,-800> qpels
mencoder: motion_est_template.c:1059: epzs_motion_search_internal: Assertion
`(P[3][0]>>shift) <= xmax' failed.

Go back to look at macroblock 73,17:

ff_estimate_b_frame_motion: estimate b forward
ff_estimate_motion_b: 73,17 P_TOPRIGHT is <-384,-288> qpels
sab_diamond_search: minima 0 is 0,0 7371 from map eb800000 = 3ae 0 0, key eba00400 = 3ae 400 400
sab_diamond_search: minima 1 is 1023,-200 4739 from map eb79c3ff = 3ad 738 3ff, key eb99c7ff = 3ae 338 7ff
sab_diamond_search: minima 2 is -96,-73 6976 from map eb7db7a0 = 3ad 7b6 7a0, key eb9dbba0 = 3ae 3b7 3a0
sab_diamond_search: minima 3 is -97,-72 6999 from map eb7dbf9f = 3ad 7b7 79f, key eb9dc39f = 3ae 3b8 39f
sab_diamond_search: minima 4 is -96,-72 6991 from map eb7dbfa0 = 3ad 7b7 7a0, key eb9dc3a0 = 3ae 3b8 3a0
sab_diamond_search: minima 5 is -95,-72 6983 from map eb7dbfa1 = 3ad 7b7 7a1, key eb9dc3a1 = 3ae 3b8 3a1
sab_diamond_search: minima 6 is -89,-72 6935 from map eb7dbfa7 = 3ad 7b7 7a7, key eb9dc3a7 = 3ae 3b8 3a7
sab_diamond_search: minima 7 is -96,-71 7018 from map eb7dc7a0 = 3ad 7b8 7a0, key eb9dcba0 = 3ae 3b9 3a0
sab_diamond_search: result is <1023,-200> pels, allowed range is -1184..112, -288..448
ff_estimate_motion_b: 73,17 got <1023,-200> pels from ezps
ff_estimate_motion_b: 73,17 got <4092,-800> qpels from sub motion

I've yet to track back where the (MotionEstContext*)c->map[1] value of
0xeb79c3ff, which is out of range, comes from.  Is the code in
sab_diamon_search supposed to detect out of range MV values and discard
them?  It looks like there is code to do that, but obviously it isn't
working.  Or should an out of range value like this never be one of the
minima in sab_diamond_search in the first place?