[MPlayer-users] MPlayer benchmarking system

[Vladimir Mosgalin]

Hi Reimar Döffinger!

 On 2012.02.14 at 22:24:10 +0100, Reimar Döffinger wrote next:

> On Wed, Feb 15, 2012 at 12:56:16AM +0400, Vladimir Mosgalin wrote:
> > Or I could make more complex patch which adds configure option for
> > simple hardening (like this) without performance impact, probably
> > enabled by default, and configure option for stronger hardening with
> > possibly slight performance impact (with stack protection activated). Do
> > you think something like that will be accepted?
> 
> Depends on how ugly it is and how much benchmarking you're willing
> to do I guess :-).
> For example I think that at least on x86_64 we should be compiling
> with -fPIE and link with -fPIE -pie by default, however I've never
> had the time and motivation to properly test it.
> On 32 bit x86 I think it shouldn't be necessary to compile with
> -fPIE in order to be able to link with -pie, so we should add
> that by default, too.

I'm not sure, I'm not an expert in this stuff. From documentation it
seems like it's required to use -fPIE when linking, nothing is said
about i386 being an exception.. And according to debian reports and
measurements, -fPIE nullifies effect from -fomit-frame-pointer on i386,
possibly resulting in serious performance loss, at least on the old
platforms (http://d-sbd.alioth.debian.org/www/pax/pie.txt).

> IMHO both of these have a far higher impact on security than all
> the other options (well, stack protector might come close, but
> heap exploits have gotten too "easy" nowadays).

Well, I was thinking of only using _FORTIFY_SOURCE by default and
leaving rest for people who value security over performance, by no means
activating such option by default. Think of it as
--disable-hardening       Disable all hardening options during compilation
--enable-extra-hardening  Enable extra hardening options, possibly leading to slowdowns
configure options


As for benchmarks of various options. Is there any suite for
benchmarking mplayer? All I've seen so far is very simple scripts
(I wrote myself a few, too), but to measure impact of changes like
compiler options or compiler versions one has to have something more
serious, in order not to spend too much time doing it by hand. I'm
thinking of something that takes bunch of various media files as input,
checking common cases like mpeg2, mpeg4, h264 video and few audio
codecs, with videos in SD, HD and "very high bitrate HD" resolutions and
stereo/multichannel audio and runs through bunch of benchmarks,
measuring separately speed of audio, video and in combination. Video
should be checked through null vo and through one or two meaningful ones
like x11 and gl. Idk what else, maybe separately check single and
multithreaded decoding and add some conversion or scaling with "scale"
video filter as extra tests. Of course, it all should be run through
several times, then summarize numbers into some table for future
comparison.
Now having something *that* serious one could really check impact of
compiler options and be more or less sure in results. I know that all I
described above sounds like a job for "a simple shell script", however
I'm quite tired of using bits and pieces of such custom-made scripts
myself and wonder if there is known more or less complete system for
benchmarking mplayer. People do seem to benchmark ffmpeg with ease, so
maybe someone made something for mplayer too.

I strongly believe that if one to seriously measure impact of every
hardening option for mplayer, scripts for what I described above must be
completed before even trying that.. At very least, I don't feel like
trying to benchmark impact of these options without complete and
automatic benchmarking system. Maybe I should help in creating that
system, there is nothing too complicated in it but I'm not sure I have
time right now (I'll think about it). I hope someone might have created
this system already.

-- 

Vladimir