[FFmpeg-devel] [PATCH] Check malloc values in swscale.

[Ramiro Polla]

On Wed, Aug 26, 2009 at 11:05 PM, Michael Niedermayer<michaelni at gmx.at> wrote:
> On Wed, Aug 26, 2009 at 05:43:45PM -0300, Ramiro Polla wrote:
>> On Sun, Aug 23, 2009 at 7:05 PM, Michael Niedermayer<michaelni at gmx.at> wrote:
>> > On Sun, Aug 23, 2009 at 06:50:09PM -0300, Ramiro Polla wrote:
>> [...]
>> >> @@ -1737,7 +1738,7 @@
>> >>
>> >> ? ? ?// Note the +1 is for the MMX scaler which reads over the end
>> >> ? ? ?/* align at 16 for AltiVec (needed by hScale_altivec_real) */
>> >> - ? ?*outFilter= av_mallocz(*outFilterSize*(dstW+1)*sizeof(int16_t));
>> >> + ? ?CHECKED_ALLOCZ(*outFilter, *outFilterSize*(dstW+1)*sizeof(int16_t));
>> >>
>> >> ? ? ?/* normalize & store in outFilter */
>> >> ? ? ?for (i=0; i<dstW; i++) {
>> >> @@ -1764,7 +1765,7 @@
>> >> ? ? ?}
>> >>
>> >> ? ? ?ret=0;
>> >> -error:
>> >> +fail:
>> >> ? ? ?av_free(filter);
>> >> ? ? ?av_free(filter2);
>> >> ? ? ?return ret;
>> >
>> > i doubt, this with the lack of checks of the return value wil work
>>
>> Do you mean that the return value of initFilter() is not checked?
>> Patch attached.
>
>> ?swscale.c | ? 20 ++++++++++++--------
>> ?1 file changed, 12 insertions(+), 8 deletions(-)
>> 03f034dc778772549431ead6bb9bd67aa41ecab0 ?initFilter_check_return.diff
>> --- swscale.c.orig ? ?2009-08-26 17:37:15.000000000 -0300
>> +++ swscale.c 2009-08-26 17:38:10.000000000 -0300
>> @@ -2812,14 +2812,16 @@
>> ? ? ? ? ? ? ?(flags & SWS_CPU_CAPS_ALTIVEC) ? 8 :
>> ? ? ? ? ? ? ?1;
>>
>> - ? ? ? ?initFilter(&c->hLumFilter, &c->hLumFilterPos, &c->hLumFilterSize, c->lumXInc,
>> + ? ? ? ?if (initFilter(&c->hLumFilter, &c->hLumFilterPos, &c->hLumFilterSize, c->lumXInc,
>> ? ? ? ? ? ? ? ? ? ? srcW ? ? ?, ? ? ? dstW, filterAlign, 1<<14,
>> ? ? ? ? ? ? ? ? ? ? (flags&SWS_BICUBLIN) ? (flags|SWS_BICUBIC) ?: flags,
>> - ? ? ? ? ? ? ? ? ? srcFilter->lumH, dstFilter->lumH, c->param);
>> - ? ? ? ?initFilter(&c->hChrFilter, &c->hChrFilterPos, &c->hChrFilterSize, c->chrXInc,
>> + ? ? ? ? ? ? ? ? ? srcFilter->lumH, dstFilter->lumH, c->param) == -1)
>
> i prefer <0 checks especially when the return value might be ENOMEM
>
> besides this, its ok if tested (manually forced malloc fail under valgrind)

All applied.