[FFmpeg-user] Chroma subsampling formats

[Mark Filipak]

Hi, All,

I have the final message to Dr. Charles Poynton for your review.
However, there's one feature that I can't confirm on my own: Does a zero
(e.g. in "4:4:0") actually indicate that a chroma picture (plane) is void?

I think that chroma subpictures (planes) can be void (i.e.
non-existent), but I have no way to parse it out from my existing sample
videos -- I don't know how to parse for subpicture planes -- and I have
no tools that will help me to confirm it.

In other words, does yuv420 even have chroma planes in its bottom field?
It doesn't need bottom-field chroma planes and can't use bottom-field
chroma planes, but does it nonetheless have bottom-field chroma planes?
After all, I reason: What would be the point of 4:2:0 pixel thinning if
all the chroma subpicture planes are populated anyways, eh?

I just need someone who knows _for_sure_ to verify it.

Thanks--Mark

=====

Dear Dr. Poynton,

Thank you for providing valuable guidance to so many people for so many
years.

You may know that Wikipedia contains a rather tortured description of
chroma subsampling. I seek to simplify and regularize the description
and to expand the scope to include the four notations that I label
"conjecture", below. Specifically, I believe that in "4:4:0" and "4:2:0"
and "4:0:0", the zeroes merely indicate that the associated chroma
subpictures are void, and that the inconsistencies noted in Wikipedia
therefore do not exist.

I'm associated with the ffmpeg-user mailing list, and though I speak
solely for myself, I have the entirety of ffmpeg as an audience. I would
be delighted to receive the opinions and experience of an authority. All
comments are welcome.

If you reply and if I desire to quote you, I promise to tell you what I
plan to quote, I promise to tell you where I plan to quote you, and I
promise to tell you the context. In other words, you must expressly give
informed consent or I will not disclose your reply and will use the
information you share solely to revise the documentation below without
approbation. Note however that I have posted drafts of this message on
ffmpeg-user, prompting for comments.

Q1. I show four conjectures: 4:4:2 and 4:4:1 and 4:2:1 and 4:0:0. In
your judgment, do those conjectured notations exist? Are they correct?

Q2. For 4:2:2 and 4:2:0 and 4:1:1, I see a difference between your
descriptions and Wikipedia's. As I see it, you show chroma samples that
are 1/2 and 1 1/2 pixels offset to the left. Do my diagrams correctly
depict your judgment?

Thank you. I hope you respond but I also wish to not burden you.

Warm Regards,
Mark Filipak.

References.
Poynton: https://www.poynton.ca/PDFs/Chroma_subsampling_notation.pdf
QuestTel: https://questtel.com/wiki/chroma-sub-mapping-types
Wikipedia: https://en.wikipedia.org/wiki/Chroma_subsampling

KEY:
:    •       : a dot symbolizes a luma pixel coinciding its luma sample.
:    • •     : 2 dots symbolize 2 separate luma pixels and samples.
:    • • • • : 4 dots symbolize 4 separate luma pixels and samples.
:    ©       : @ symbolizes a chroma pixel coinciding its chroma sample.
:    © =     : as above, plus the chroma pixel is copied to the right.
:    © = = = : as above, plus 2 more chroma pixel copies, all identical.
:   ©= =     : @ is a chroma sample acquired 1/2 pixel to the right.
:   ©= =©= = : 2 of the above: '©= =' and '©= =', side by side.
:   ©= = = = : a chroma sample acquired 1/2 pixel to the right.
: ©  = = = = : as above except that the gap is 1 1/2 pixels wide.
:    ——[1]—— : asserts that there is no chroma subpicture present [1].

Shown next is the top-left corner of a sampled film frame.
+—————————
| • • • •    <== Four consecutive pixels in line 0 of subpicture-0.
| • • • •    <== Four consecutive pixels in line 0 of subpicture-1.

Shown next is the loci of Y, Cb, & Cr separated as element 'planes'.
|    Y            Cb           Cr
| • • • •    | © © © ©    | © © © ©    4:4:4, FFmpeg, Poynton,
| • • • •    | © © © ©    | © © © ©    Wikipedia.

| • • • •    | © © © ©    | © © © ©    4:4:2, conjecture.
| • • • •    | © = © =    | © = © =

| • • • •    | © © © ©    | © © © ©    4:4:1, conjecture.
| • • • •    | © = = =    | © = = =

| • • • •    | © © © ©    | © © © ©    4:4:0, FFmpeg, Wikipedia.
| • • • •    | ——[1]——    | ——[1]——

| • • • •    | © = © =    | © = © =    4:2:2, FFmpeg, Wikipedia.
| • • • •    | © = © =    | © = © =

| • • • •    |©= =©= =    |©= =©= =    4:2:2 (BT.601), FFmpeg, Poynton.
| • • • •    |©= =©= =    |©= =©= =    [2]

| • • • •    | © = © =    | © = © =    4:2:1, conjecture.
| • • • •    | © = = =    | © = = =

| • • • •    | © = = =    | © = © =    4:2:1, QuestTel (vaguely
| • • • •    | © = = =    | © = © =    described, not pictured).

| • • • •    | © = © =    | © = © =    4:2:1 (JPEG variant, irregular),
| • • • •    | ——[1]——    | © = = =    Wikipedia.

| • • • •    | © = © =    | © = © =    4:2:0 (MPEG1, H.261), FFmpeg,
| • • • •    | ——[1]——    | ——[1]——    Poynton, Wikipedia.

| • • • •    |©= =©= =    |©= =©= =    4:2:0 (MPEG2 frames), FFmpeg,
| • • • •    | ——[1]——    | ——[1]——    Poynton, QuestTel. [2]

| • • • •    | © = = =    | © = = =    4:1:1, FFmpeg, QuestTel,
| • • • •    | © = = =    | © = = =    Wikipedia.

| • • • •   ©  = = = =    |  ©= = = =  4:1:1 (480i), FFmpeg, Poynton.
| • • • •   ©  = = = =    |  ©= = = =  [2]

| • • • •    | © = = =    | © = = =    4:1:0, FFmpeg.
| • • • •    | ——[1]——    | ——[1]——

| • • • •    | ——[1]——    | ——[1]——    4:0:0, (monochrome) conjecture.
| • • • •    | ——[1]——    | ——[1]——

| • • •      | © = =      | © = =      3:1:1 (Sony), Poynton,
| • • •      | © = =      | © = =      Wikipedia.

[1] The chroma subpicture is void. If both chroma subpictures-0 & -1 are
void (e.g. 4:0:0), then there are no chroma subpictures & the composited
pixels carry luma only. Otherwise, the decoder copies chroma
subpicture-0 to chroma subpicture-1. Note that though it is possible, a
condition in which chroma subpicture-0 is void & chroma subpicture-1 is
populated has not been seen in the wild.

[2] The chroma samples are shown left-shifted by one-half pixel,
presumably due to sampling one-half pixel to the left. Since pixels
cannot be moved, their values must be derived, usually by mixing
samples. Note that mixing is most accurately accomplished in frame
pictures and that the left edges of chroma spans become slightly blurry.
Note also that chroma spans are assumed to be truncated as needed at the
left edge of film frames, e.g. in 4:1:1.