[FFmpeg-devel] Is this the expected behavior in libswscale/input.c

[Dan Parrot]

On Thu, 2016-06-09 at 17:01 -0400, Ronald S. Bultje wrote:
> Hi,
> 
> On Thu, Jun 9, 2016 at 4:02 PM, Dan Parrot <dan.parrot at mail.com> wrote:
> 
> > Line 72 of libswscale/input.c is:
> > dstU[i] = (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >>
> > RGB2YUV_SHIFT;
> >
> > The definition of macro RGB2YUV_SHIFT in libswscale/swscale_internal.h
> > is on line 417:
> > #define RGB2YUV_SHIFT 15
> >
> > By examining the result of executing line 72 in input.c it appears that
> > the radix used for macro RGB2YUV_SHIFT is hexadecimal. So that 0x10001
> > is shifted left by 20 in subexpression 0x10001<<(RGB2YUV_SHIFT-1) with
> > the result being 0x100000.
> >
> > I was expecting a left-shift of 14, given the macro definition.
> >
> > Should the macro be interpreted as decimal 15 or hexadecimal 15?
> 
> 
> It's decimal 15, it should shift by 14. Check disassembly, though... It's
> almost impossible that your compiler would read this as hex and still be
> able to produce runnable applications. Maybe submit a bug report to your
> compiler's bug tracker?
> 
> Ronald
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

There is no issue with the software. I have been examining the code in
gdb and what confused me was having changed the default gdb radix with
"set radix 16".

Thanks Ronald.