[FFmpeg-devel] [PATCH 3/5] avcodec/proresenc_kostya: do not write into alpha reserved bitfields

[Clément Bœsch]

On Mon, Jan 08, 2024 at 09:10:09PM +0100, Stefano Sabatini wrote:
> On date Sunday 2024-01-07 19:16:45 +0100, Clément Bœsch wrote:
> > This byte represents 4 reserved bits followed by 4 alpha_channel_type bits.
> > 
> > alpha_channel_type currently has 3 differents defined values: 0 (no
> > alpha), 1 (8b alpha), and 2 (16b alpha), all the other values are
> > reserved. This part is correctly written (alpha_bits>>3 does the correct
> > thing), but the 4 initial bits are reserved.
> > ---
> >  libavcodec/proresenc_kostya.c          | 2 +-
> >  tests/ref/vsynth/vsynth1-prores_ks     | 2 +-
> >  tests/ref/vsynth/vsynth2-prores_ks     | 2 +-
> >  tests/ref/vsynth/vsynth3-prores_ks     | 2 +-
> >  tests/ref/vsynth/vsynth_lena-prores_ks | 2 +-
> >  5 files changed, 5 insertions(+), 5 deletions(-)
> > 
> > diff --git a/libavcodec/proresenc_kostya.c b/libavcodec/proresenc_kostya.c
> > index de63127192..f6c71c2b86 100644
> > --- a/libavcodec/proresenc_kostya.c
> > +++ b/libavcodec/proresenc_kostya.c
> > @@ -1051,7 +1051,7 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt,
> >      bytestream_put_byte  (&buf, pic->color_primaries);
> >      bytestream_put_byte  (&buf, pic->color_trc);
> >      bytestream_put_byte  (&buf, pic->colorspace);
> > -    bytestream_put_byte  (&buf, 0x40 | (ctx->alpha_bits >> 3));
> > +    bytestream_put_byte  (&buf, ctx->alpha_bits >> 3);
> 
> Shall be good, I wonder why it was done that way (probably there was
> no open specification at the time?).

The Apple encoder tends to write stuff in this reserved area. Currently on
the system I'm testing VideoToolBox is actually writing 0xE in the high
nibble (instead of 0x4 here). So far it doesn't seem to have any impact (I
tried to encode mov/prores files with and without alpha, with both ks and
aw encoder, after this patchset, and the files play just fine with
QuickTime).

I have no idea what impact it has internally as I was unable to locate the
encoder or the decoder in the Apple core frameworks.

-- 
Clément B.