[FFmpeg-devel] SAMPLE_FMT_FLT bounds

[Vitor Sessak]

Colin McQuillan wrote:
> I'm looking for the correct way to clip the output of a floating-point decoder.
> 
> audioconvert.c converts floats thus:
> lrintf(*(const float*)pi * (1<<15))) to convert to int16_t
> lrintf(*(const float*)pi * (1<<7)) + 0x80) to convert to uint8_t
> 
> qcelpdec.c clips to [-1.0, 32767.0/32768.0]. This is alright when
> converting to 16-bit but will overflow when converting to 8-bit.
> Should I clip to [-1, 127.0/128.0]? Or [-128.499/127, 127.499/128]?

I think that the attached makes more sense. All formats we have that 
output SAMPLE_FMT_FLT either output real floats (like pcm_flt) or were 
designed to be converted to S16.

-Vitor
-------------- next part --------------
A non-text attachment was scrubbed...
Name: audioconvert.diff
Type: text/x-diff
Size: 1529 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090806/2f751b48/attachment.diff>