[FFmpeg-devel] Broken endian indication in pixfmt list

[wm4]

The header pixfmt.h contains the following comment in the pixel format
list doxygen:

 * @note
 * Make sure that all newly added big-endian formats have (pix_fmt & 1) == 1
 * and that all newly added little-endian formats have (pix_fmt & 1) == 0.
 * This allows simpler detection of big vs little-endian.

This is currently broken for the following formats: nv20le/be gbrap16le/be

Obviously, this can't be fixed until the next ABI bump. But more
importantly, this looks like a stupid convention, and doesn't even
allow you to distinguish formats that need endian swapping and those
that don't.

Wouldn't it be better to have a flag that indicates whether the format
is endian-independent? I would define the flag as follows: if there is
a swapped-endian format defined, both formats have the flag set. I
think this would be useful to the API user.

(You can create this flag yourself with the currently available API and
without using the LSB-trick mentioned above, but it's rather painful.
For example, RGB444 can be accessed in an endian-independent way
according to its pixdesc, and av_read_image_line() does only 1 byte
reads for this format, because the components are neatly aligned.)

Should I write a patch that adds a AV_PIX_FMT_FLAG_ENDIAN flag?

Also, we should discuss whether swapped-endian formats should even
exist. From what I can see, only the raw decoders/encoders make
effective use of them. Wouldn't it be simpler to let the
decoders/encoders do the byte-swapping?