[FFmpeg-devel] lurking bugs in the mmx-related assembler code (?)

[u-h8zb at aetey.se]

Hello,

Would someone familiar with the mmx assembler parts of ffmpeg
look over and check that the fp-state is restored as needed?

This is crucial with musl libc which uses floating point in its
malloc() implementation.

My troubleshooting strongly suggests that it is the mmx code which is
the culprit of ffmpeg misbehavior (the same issue has been recently
fixed in libtheora).

Ffmpeg built against musl behaves erratically on certain combinations of
inputs and outputs, which may also depend on other settings.
Most often ffmpeg hangs forever quite early while beginning the conversion
(at "frame=    0 fps...." or even before showing this) but sometimes it
segfaults at that point or while being manually interrupted.

At least the internal theora decoder is affected but there may be other
misbehaving decoders or encoders. I think I observed similar breakage with
other file types than ogg/theora but did not document all the tested
combinations.

Affected ffmpeg versions: tested 2.8.8, 3.0.2, 3.1.3, same behaviour,
also when building with various gcc versions (4.x, 5.x, 6.x).

Disabling assembler in ffmpeg gets rid of the problem.

Given patches/fixes I will be glad to test them and report here.

Regards,
Rune
P.S. an attempt to report via trac did not succeed, that's why posting
to the list