[FFmpeg-devel] [RFC] H.264/SQV3 separation: h264data.h

[Diego Biurrun]

On Sun, Dec 14, 2008 at 05:03:04PM +0100, Michael Niedermayer wrote:
> On Sun, Dec 14, 2008 at 04:24:43PM +0100, Diego Biurrun wrote:
> > On Mon, Dec 01, 2008 at 02:02:57PM +0100, Michael Niedermayer wrote:
> > > On Mon, Dec 01, 2008 at 11:25:59AM +0100, Diego Biurrun wrote:
> > > > On Sat, Nov 29, 2008 at 05:28:27PM +0100, Michael Niedermayer wrote:
> > > > > On Fri, Nov 28, 2008 at 09:46:19PM +0100, Diego Biurrun wrote:
> > > > > > On Fri, Nov 28, 2008 at 07:14:57PM +0100, Michael Niedermayer wrote:
> > > > > > > On Fri, Nov 28, 2008 at 05:48:50PM +0100, Diego Biurrun wrote:
> > > > > > > > Will this need to be benchmarked at some point?
> > > > > > > 
> > > > > > > id say it cant hurt to benchmark it before commit
> > > > > > 
> > > > > > Hmmmmm
> > > > > > 
> > > > > > I ran the following benchmarks on my K6-III with two different samples:
> > > > > > 
> > > > > > http://samples.mplayerhq.hu/V-codecs/h264/cathedral-beta2-400extra-crop-avc.mp4
> > > > > > http://samples.mplayerhq.hu/V-codecs/h264/Aladin.mpg
> > > > > > 
> > > > > > I used the following MPlayer command lines to benchmark:
> > > > > > 
> > > > > > mplayer -quiet -frames 1000 -benchmark -nosound -vo xmga
> > > > > > mplayer -quiet -frames 1000 -benchmark -nosound -vo null
> > > > > > 
> > > > > > I made five runs for each sample and vo and threw away the best and
> > > > > > worst times.  The results are attached.
> > > > > > 
> > > > > > Unfortunately there seems to be a slight slowdown with the cathedral
> > > > > > sample, with Aladin it's a bit inconclusive.
> > > > > > 
> > > > > > Now I don't know how to continue from here.  Were my benchmarking
> > > > > > procedures sensible?  (BTW, is there a good way to benchmark with ffmpeg
> > > > > > directly?)  
> > > > > 
> > > > > time ffmpeg ... >&/dev/null
> > > > 
> > > > Sure, but what options would I use?
> > > 
> > > maybe
> > > -i your_file -y -an -f null /dev/null
> > 
> > OK, I redid the benchmarks directly with ffmpeg on my K6-III, here are
> > the results of 5 runs each, highest and lowest result discarded, all
> > services stopped:
> [...]
> > ffmpeg-split -i cathedral-beta2-400extra-crop-avc.mp4 -an -f null -y -benchmark -v -1 /dev/null
> > bench: utime=284.858s
> > bench: utime=284.110s
> > bench: utime=283.610s
> > 
> > ffmpeg-vanilla -i cathedral-beta2-400extra-crop-avc.mp4 -an -f null -y -benchmark -v -1 /dev/null
> > bench: utime=283.438s
> > bench: utime=282.830s
> > bench: utime=283.170s
> > 
> > 
> > The result seems to be no slowdown or a very minor slowdown below 1%.
> > 
> > So can I commit the next part of the split and continue working on it?
> 
> no, the apparent slowdown of cathedral-beta2-400extra-crop-avc.mp4 is not ok

The slowdown is a whopping 0.37%, this is insane, differences between
compiler versions or differences from completely random changes have a
much bigger influence.  This was a topic on IRC today:

17:10 <+uau> DonDiego: btw michael's latest complaint about the 
             slowdown is completely baseless
17:11 <+uau> it's a 0.28 percent change
17:11 <@DonDiego> i was just going to run the numbers...
17:11 <+uau> if you make a few completely random changes to the code 
             you can make it speed up much more :)
17:11 <@DonDiego> suggest some :)
17:17 <+uau> i tried adding av_noinline to one randomly picked 
             function, that gave a consistent 0.6 percent speedup with 
             a particular test file
17:17 <@DonDiego> which one?
17:19 <+ssvb> uau: ... also sometimes with a specific version of the 
              compiler and specific cpu revision...
17:19 <+uau> then i tried adding a pointless global function that is 
             not called from anywhere
17:19 <+uau> that made it 1.1% slower
17:20 <+uau> probably adding a bit different unused function would make 
             it a similar amount faster
17:21 <+ssvb> there are different factors such as code alignment and 
              cache aliasing which may introduce such random 
              performance changes
17:22 <+ssvb> so if the code just shifts a bit, you may have speedup or 
              slowdown in some completely unrelated part of program
17:22 <+uau> ssvb: that was my point, michael's complaint based on 0.28 
             percent performance change on one particular sample makes 
             little sense
17:43 <+ssvb> uau: you can try to do the same benchmark with different 
              versions of gcc and running it on different cpus, then 
              consider average effect

My opinion is that the benefits of the refactoring vastly outweigh
concerns about less than 0.5% speed difference, especially if it is not
clear that the speed difference is directly related to the changes.

Diego