[FFmpeg-devel] new packed pixel formats (machine vision)

[Lynne]

On 08/10/2024 09:31, Diederick C. Niehorster wrote:
> Hi All,
> 
> I am using ffmpeg for a library to interface with machine vision
> cameras that i am developing (not yet released),it allows storing the
> streams with really nice performance directly to, e.g., x264/mp4
> (thanks ffmpeg!). For this, i am looking to support some new pixel
> formats as input formats in swscale.
> A first step for this would be to describe these formats in a
> AVPixFmtDescriptor. Example machine vision pixel formats are:
> Mono10p: 10-bit luma only: 4 pixels packed into 5 bytes (i.e., no padding)
> BayerRG10: 10-bit color components, in bayer patterns, 1 component put
> into 2 bytes (10 bits data+6 bits padding)
> BayerRG10p: 10-bit color components, in bayer pattern, 4 color
> components packed into 5 bytes (really the same packing as Mono10p).
> And also 12 bit variants.
> 
> See https://www.1stvision.com/cameras/IDS/IDS-manuals/en/basics-monochrome-pixel-formats.html
> for a diagram of the Mono10p and
> https://www.1stvision.com/cameras/IDS/IDS-manuals/en/basics-raw-bayer-pixel-formats.html
> for diagrams of the packed and not packed bayer formats.
> 
> I am wondering how to map these to AVPixFmtDescriptors.
> BayerRG10 is a problem:
>      [AV_PIX_FMT_BAYER_RGGB10] = {
>          .name = "bayer_rggb10",
>          .nb_components = 3,
>          .log2_chroma_w = 0,
>          .log2_chroma_h = 0,
>          .comp = {
>              { 0, 2, 0, 0, ? },
>              { 0, 2, 0, 0, ? },
>              { 0, 2, 0, 0, ? },
>          },
>          .flags = AV_PIX_FMT_FLAG_RGB | AV_PIX_FMT_FLAG_BAYER,
>      },
> What should be put for the component depths? for 8bit bayer, 2,4,2 is
> used and for 16bit 4,8,4. This I guess signifies that for rggb 1/4
> components is red or blue and 2/4 are green. By this logic, the values
> should be 2.5,5,2.5 for a ten-bit bayer format, which is not possible
> (12bit is possible, with 3,6,3). How could i handle this?
> 
> For
> Mono10p (would be something like gray10p)
> BayerRG10p (would be something like bayer_rggb10p)
> my first question is: should this be encoded as bitstreams since their
> pixel values are not byte-aligned?
> 
> Lastly, if i figure this out, is this something that might be
> considered for inclusion in ffmpeg, or is there a policy/strong
> opinions against these machine vision formats?
> 
> All the best,
> Dee
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".

We have support for AV_PIX_FMT_BAYER_RGGB16, since its a common Bayer 
layout that cinema cameras output, so its definitely within the scope 
and not some application-specific pixfmt.
RGGB10 is just a bitpacked version.

Unfortunately, we do not directly support 10bit packed pixel formats, 
since we can't fit them into our definition, as we only support 
byte-aligned formats.

We treat those as codecs, for example AV_CODEC_ID_V210 and
AV_CODEC_ID_BITPACKED.
The format would have to be implemented as a codec, with a decoder that 
accepts AV_CODEC_ID_RGGB10 and outputs AV_PIX_FMT_RGGB16, setting
avctx->bits_per_sample to 10 to indicate how many bits are used.

The decoder would simply call get_bits(10) and set each output pixel to 
the value it gets, nothing complex. You can see how
libavcodec/bitpacked_enc.c operates and use it as a template.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0xA2FEA5F03F034464.asc
Type: application/pgp-keys
Size: 624 bytes
Desc: OpenPGP public key
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20241008/f2e48419/attachment.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 236 bytes
Desc: OpenPGP digital signature
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20241008/f2e48419/attachment.sig>