[FFmpeg-devel] tkhd transformation matrix in mov is ignored except for width/height and scaling

[Samuel Gendler]

On Mon, Nov 2, 2009 at 3:28 AM, Daniel G. Taylor <dan at programmer-art.org>
 wrote:

>
> I believe the issue is related to not having a rotation filter in FFmpeg
> at the moment (correct me if I'm mistaken) so even if we have the data
> stored in the mov context the best that can be done is writing the same
> values to the final output.
>
> I take a slightly different approach here and use a little script I
> wrote to try and guess the rotation angle using the inverse sin/cos and
> a little Python. You aren't the first to have issues with this stuff so
> today I released it here:
>
> http://github.com/danielgtaylor/qtrotate/
>
> It's the same script I use at work to automagically detect rotation and
> then I insert the MPlayer rotate filter into our processing chain so
> that we can process rotated videos from customers (and we get a few from
> iPhones and digicams). It's far from perfect but so far it has worked
> nicely.
>

ffmpeg shouldn't require a rotation filter to handle this corectly, I'd
think.  It should render each pixels through the matrix, which will result
in rotation without any filter in the chain.  It seems like performing the
necessary math during the decode process shouldn't be too hard to add.  I
assume that the scaling info is being used during decode, and it would seem
a good bet that the full matrix transformation could be applied at the same
place to get the correct result.  The one weirdness is the inversion of
width and height in the outbound stream.

I don't think preserving the matrix from incoming to outbound quicktime
tracks is fully sufficient, unfortunately, since a file that is both rotated
and scaled will be the matrix multiple of a rotate and a scale matrix, while
the resulting file will have the had the scaling applied (there is code
already in place which explicitly pulls the scale info to be used by the
decoder, but ignores rotation), so the matrix will have to be modified for
the new file.  I put an ugly hack into my codebase, which copies the matrix
across from the incoming file to the outgoing one while preserving rotation
but dropping the scale info, but I couldn't find any context that exists
from the quicktime parser all the way through to the quicktime writer, so I
just made a massively ugly global array for the matrix and I write to it
from the parser and read it in the code which writes a new tkhd node (which
currently just has an identity matrix hardcoded into it). Obviously, files
with multiple video tracks with varying matrices will fail, but the iphone
(and ipod nano) won't generate those and, to the best of my knowledge,
that's the only device that writes a non-identity matrix into quicktime
files.

It's a kludge, but it gets me to where I need to be for now.  My only other
option is to do what you did and write raw video from ffmpeg, then run it
through rotate in mplayer before re-encoding, and that's a heck of a lot
more work, processing time, and disk i/o.  Once I finish polishing it up,
I'll stick it in an email to ffmpeg-users so at least others having the same
trouble as me will have something to get them through. It's obviously not a
'patch' that could go into the codebase, but it is useful.