[FFmpeg-devel] [PATCH] libavutil/display: improve av_display_rotation_get for reflections

[Syed AbuTalib]

The av_display_rotation_get function previously yielded inconsistent
angles for certain reflected (mirrored) matrices. For instance,
a matrix representing a -45 degree rotation followed by a horizontal
flip resulted in 135 degrees, while the desired interpretation after
canonicalizing the reflection is 45 degrees.

This patch revises av_display_rotation_get to:
1. Detect reflections by checking the determinant of the 2x2 transform.
2. If a reflection is detected, it canonicalizes the matrix by effectively
   negating the components derived from the first column (m0, m3)
   before calculating the atan2 arguments. This is equivalent to
   extracting the rotation from M * F_h (where F_h is a horizontal flip).
3. The final negation in the return path is preserved.

This change ensures more predictable rotation angle extraction when
reflections are involved. For example, a pure horizontal flip now
correctly reports 0 degrees.

The FATE test in libavutil/tests/display.c has been updated:
- The original test sequence now has assertions checking against the
  new, correct behavior of the patched function.
- An additional 8 comprehensive test cases have been added to cover
  0, 90, 180, 270 degree rotations, both unflipped and when
  composed with a horizontal flip.
- The FATE reference file for 'fate-display' has been updated to
  reflect these changes and new test outputs.
All tests now pass with this revised logic.

Signed-off-by: Syed AbuTalib <lowkey at google.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-libavutil-display-improve-av_display_rotation_get-fo.patch
Type: application/octet-stream
Size: 9986 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20250604/c986d694/attachment.obj>