[FFmpeg-devel] [libav-devel] [RFC] M-bit to N-bit YUV conversions

[Ronald S. Bultje]

Hi Oskar,

On Thu, Aug 25, 2011 at 2:59 AM, Oskar Arvidsson <oskar at irock.se> wrote:
> On Wednesday 24 August 2011 18.55.10 Ronald S. Bultje wrote:
>> [...]
>>
>> Dither is all just because of ordered dither and I don't feel it's
>> worth looking into. I've been planning to replace it with
>> Atkinson-style dither (that's like f-s, which x264 uses) so let's just
>> work towards that and be done with it.
>
> Still, ordered dither would actually work as intended if only using simple
> shifting.

Right, but simple shifting changes range.

> Also, I think the underlying problem would be the same with any type
> of dither as what the current conversion method does is to increase the values
> - thus posing larger errors in e.g. error diffusion if not the "byte
> duplication" is reversed in one or another way.

Correct. I think the correct solution is that for downscaling (e.g.
10-to-8-bit), we need to undo the byte duplication to correct the
range again. So if for luma, 0x80 is upped to 0x201 in 8-to-10-bit
conversion), then the reverse should be done also. I told you I intend
hscale to use 1,0000,0000,0100,00b sum for upscaling to maintain
range. The reverse could be done by using a sum of slightly less than
0x4000 (I don't know exactly how much right now, but basically 0x4000
* 0x400 / 0x401, just like the above sum is 0x4000 * 0x401 / 0x400).
That would undo the byte duplication and change 0x201 back to 0x80 for
luma. Does that sound acceptable? It's also easy to code up.

>> > b) Should we distinguish between limited and full range YUV?
>>
>> Doesn't convertJpegRange already do that? Are you afraid that we
>> become "out of range" when we scale between bitdepths with different
>> ranges? (I admit I haven't looked much into that yet, i.e. don't know
>> how to handle it yet.)
>
> I'm not too worried about becoming out of range, but the fact that to my
> knowledge there's no distinction between limited and full range YUV in swscale
> at the moment - having different methods for these would require more work and
> would also mean we have to discuss user options etc.

I think more and more that hscale coeff sum can help here, plus an
additional hscale coeff base.

Ronald