[FFmpeg-devel] [RFC] Exporting the alpha mode from decoders

[wm4]

This is a proposal for an API extension.

Currently, some pixel formats support alpha, but whether the alpha
component contains something useful or just garbage is not part of the
pixel format definition. This applies at least to packed RGB formats,
where the 4th component is either alpha or garbage depending on the
context.

This means that if a decoder outputs such a packed RGB format, an API
user has no idea whether it has alpha or not. E.g. take PNG and
Camtasia (tscc.c). PNG with alpha outputs AV_PIX_FMT_RGBA, and Camtasia
can output AV_PIX_FMT_RGB32 (which is ARGB or BGRA depending on
endian). Camtasia supports no alpha, and the alpha component literally
contains garbage (AFAIK and judging by the single sample I've seen). An
application decoding both of these can't know whether the output frame
has alpha or not, unless every codec is special-cased.

One possible solution is duplicating all these pixel formats, so you'd
have e.g. AV_PIX_FMT_RGBA and AV_PIX_FMT_RGBX. But I think we all agree
that we don't want more pixel formats.

The other solution, which I want to advocate here, is adding a field to
AVFrame that indicates the alpha mode. Something like:

typedef enum AVAlphaMode {
    // if an alpha component is present, its function is unknown, and
    // it may be garbage
    AV_ALPHA_UNKNOWN,
    // the alpha component contains premultiplied alpha
    // (not sure if any file format actually uses this)
    AV_ALPHA_PREMULTIPLIED,
    // the alpha component contains straight alpha
    // (e.g. PNG)
    AV_ALPHA_STRAIGHT,
} AVAlphaMode;

typedef struct AVFrame {
    ...
    AVAlphaMode alpha_mode;
} AVFrame;

Thoughts?