[FFmpeg-devel] [PATCH] videodsp: don't overread edges in vfix3 emu_edge.

[Andreas Cadhalpun]

Hi Ronald,

On 24.10.2015 17:55, Ronald S. Bultje wrote:
> On Sat, Oct 24, 2015 at 11:41 AM, Andreas Cadhalpun <
> andreas.cadhalpun at googlemail.com> wrote:
> 
>> On 24.10.2015 14:47, Ronald S. Bultje wrote:
>>> ---
>>>  libavcodec/x86/videodsp.asm | 15 +++++++++++----
>>>  1 file changed, 11 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/libavcodec/x86/videodsp.asm b/libavcodec/x86/videodsp.asm
>>> index 25d4364..48f5ac0 100644
>>> --- a/libavcodec/x86/videodsp.asm
>>> +++ b/libavcodec/x86/videodsp.asm
>>> @@ -194,8 +194,12 @@ hvar_fn
>>>  %elif (%2-%%off) == 2
>>>      mov            valw, [srcq+%2-2]
>>>  %elifidn %1, body
>>> -    mov            vald, [srcq+%2-3]
>>> -%else
>>> +    mov            valb, [srcq+%2-1]
>>> +    sal            vald, 16
>>> +    mov            valw, [srcq+%2-3]
>>> +%elifidn %1, bottom
>>> +    movd mm %+ %%mmx_idx, [srcq+%2-4]
>>> +%else ; top
>>>      movd mm %+ %%mmx_idx, [srcq+%2-3]
>>>  %endif
>>>  %endif ; (%2-%%off) >= 1
>>> @@ -251,12 +255,15 @@ hvar_fn
>>>      mov     [dstq+%2-2], valw
>>>  %elifidn %1, body
>>>      mov     [dstq+%2-3], valw
>>> -    shr            vald, 16
>>> +    sar            vald, 16
>>>      mov     [dstq+%2-1], valb
>>>  %else
>>>      movd           vald, mm %+ %%mmx_idx
>>> +%ifidn %1, bottom
>>> +    sar            vald, 8
>>> +%endif
>>>      mov     [dstq+%2-3], valw
>>> -    shr            vald, 16
>>> +    sar            vald, 16
>>>      mov     [dstq+%2-1], valb
>>>  %endif
>>>  %endif ; (%2-%%off) >= 1
>>>
>>
>> This fixes the crash and passes FATE, so looks good to me.
>> Maybe mention the bug report in the commit message, e.g.:
>> Bug-Debian: https://bugs.debian.org/801745
> 
> 
> Thanks for testing, so this does actually fix the bug?

Yes, at least I couldn't reproduce the crash anymore with this patch applied.

Best regards,
Andreas