[FFmpeg-devel] [PATCH] libavutil: add clean aperture (CLAP) side data.

[Neil Birkbeck]

On Mon, Apr 27, 2020 at 4:38 AM Lynne <dev at lynne.ee> wrote:

> Apr 27, 2020, 05:27 by neil.birkbeck at gmail.com:
>
> > The clean aperature represents a cropping of the stored image data used
> to
> > relate the image data to a canonical video system and exists as container
> > metadata (see 'clap' section in
> https://developer.apple.com/library/archive/documentation/QuickTime/QTFF/QTFFChap3/qtff3.html
> )
> >
> > Addition of the side data is a first step towards demuxing CLAP atom
> metadata,
> > helping to resolve https://trac.ffmpeg.org/ticket/7437
> >
> > This CleanAperture representation can also carry PixelCrop fields from
> MKV.
> > Side data was suggested as a way to carry such PixelCrop fields in:
> > https://ffmpeg.org/pipermail/ffmpeg-devel/2016-March/192302.html
> >
> > Transmuxing the side data can then be added (MOV to/from MKV), and
> > auto-application could optionally be enabled like autorotate in
> ffmpeg_filter.c.
> >
>
> We already have frame cropping metadata in the AVFrame structure.
>

Propagating the crop fields from the format/container to AVFrame was my
initial thought. As far as I can tell, currently the cropping fields on
AVFrame are typically set from the coded bitstream. And plumbing from the
container requires the crop fields to be added to AVCodecParameters and to
AVCodecContext, which could then be set on the AVFrame in decode.c.

In part, I based the SideData on some older threads on the topic, but those
threads predate the AVFrame-level cropping. I still see a few a couple of
other potential reasons:
-For the transmux use cases, I was concerned adding the extra fields within
AVCodecParameters/AVCodecContext may confuse the interfaces as to whether
crop/CLAP was destined for the container or the codec. As stream metadata,
it seemed more clear that it should belong in the container
-And a minor one: for the CLAP atom case, on remuxing, the existing integer
crop fields could cause some small sub-pixel loss.

Is it preferred to add the extra fields in AVCodecParameters/AVCodecContext
and plumb them into the frame instead? That should work for the case that
I'm trying to resolve (applying PixelCrop from MKV), but wanted to make
sure the other use cases (e.g., transmux, CLAP atom) worked too.