[FFmpeg-devel] [Patch][GSoC] Motion Estimation filter

[Thilo Borgmann]

Am 13.04.16 um 22:01 schrieb Michael Niedermayer:
> On Wed, Apr 13, 2016 at 07:24:42PM +0000, Davinder Singh wrote:
>> good vectors? how can i improve them? since it search for every possible
>> place, it should give best match. can you give more details, why
>> surrounding vectors need to be considered?
> 
> the goal is to find the true motion that is how the physical object
> moves, the best match pixel by pixel may not represent the true motion
> at all
> for example a block that contains a finely detailed high contrast
> object and that matches with a low difference to only one spot in the
> new image would likely represent a good/true motion vector
> a block that contains a sharp straight line and that matches well
> would likely be accurate in the direction normal to the line but may
> be somewhat inaccurate in the direction along the line
> a block containing nearly flat constant color will quite possibly
> match somewhere best that is very far from the true motion
> and blocks that on another frame move off screen or behind other objects
> will also match best somewhere pixel by pixel but wherever they will
> match it will not be the true motions direction
> 
> whats needed is to have the "strong/good" matching vectors pull any
> surrounding weak ones towards themselfs in some form.
> 
> Researching algorithms/papers that attempt to find true motion well and
> robustly, may be needed. Its liklely theres some existing research and
> experiments in that area. Of course we will have to experiment too
> but there should be some existing stuff

I don't know the context about this, so I just throwing in my
thoughts... maybe they are valuable, maybe not.

For a qualification task, the robustness of the "true" motion can easily
be enhanced a lot by two simple things:
First, do the search in an image pyramid (if this is not yet included)
and descend towards your minimum matching cost while upscaling to the
original resolution.
Second, do a forward-backward consistency check. Robust vectors are the
ones where MV_i == -(MV_i+1). Define a threshold of 1, 1/2 or 1/4 pixel
to discard bad matches.
If the filter has to yield a dense vector field, good matches have to be
extrapolated. There are also several approaches for this task...

Beyond a qualification task, this basic scheme could later be reused
with more sophisticated algorithms than simple block matching. Like
cross based matching and other block comparison metrices. Also works
with more sophisticated search algorithms like plane sweeping.

In the long run there should also be a global approach not solely based
on blocks which in general yield most exact results with most complexity.

Determining "true motion" correlates very closely to 3D reconstruction
where depth is determined by "true motion".
You find a comprehensive overview within stereo and multi-view
benchmarks, like provided by Middleburry. Google for "Middleburry
vision" to find the papers.

Another closely correlating are would be "optical flow".

In both areas there are quite fast algorithms available.


> Its important to note though that this problem differs from motion
> estimation used in video encoding.
> But the methods used in video encoding still should perform much better
> than "flat" independant exhaustive search (they are also MUCH faster)
> For encoder targeted ME there is UMH and the older EPZS
> i dont know what algorithm is good for finding true motion
> but the encoder targeted ones at least have a term that minimizes
> vector differnces (as there is cost in a encoder to store the
> difference from the predicted)
> 
> 
> 
>>
>> also i tried to compare it with export_mvs, they seems quite similar.
>> the export_mvs ones, sometimes use multiple vectors at for certain blocks.
>>
>>
>> DSM_
>>
>> On Tue, Apr 12, 2016 at 5:44 AM Michael Niedermayer <michael at niedermayer.cc>
>> wrote:
>>
>>> On Sat, Apr 09, 2016 at 11:50:07PM +0000, Davinder Singh wrote:
>>>> ok, applied the changes. with new closest vector condition. will add
>>>> threshold now.
>>>
>>> it may be beyond the scope of this qualification task but to get
>>> good quality vectors the surrounding vectors will have to be
>>> considered in choosing each vector
>>>
>>> [...]

-Thilo