[FFmpeg-devel] MPEG-2 FLAGS2_FAST benchmarks

[Michael Niedermayer]

On Sat, Jun 13, 2020 at 05:11:05PM +0100, Kieran Kunhya wrote:
> Hello,
> 
> I have run the following commands on the following CPU which is 7 years old:
> Intel(R) Xeon(R) CPU E3-1265L v3
> 
> ffmpeg_g -i matrixbench_mpeg2.mpg -f null -
> and
> ffmpeg_g -flags2 fast -i matrixbench_mpeg2.mpg -f null -
> 
> Here are the results:
> 
> Normal Fast
> 2202 2168
> 2155 2057
> 2210 2128
> 2104 2051
> 2132 2053
> 2035 2014
> 2021 2037
> 2247 1999
> 2095 2015
> 2232 2119
> Mean 2143.3 2064.1
> Stdev 79.82209385 55.9075626
> 
> It can therefore be shown that the difference is statistically
> insignificant.
> 
> I would like to propose removing MPEG-2 fast mode.
> Removing it also saves ~4KB on the stripped ffmpeg binary.

I would have preferred not to have to defend my code contributions
repeatedly. Something which seems unevenly requested from developers.
But then whatever, writing this here also is a useful and informative
mail i hope. 

Lets go through it, each problem alone more or less invalidates the claim IMO.

Problem 1
you have a mean of around 2100 and Stdev of about between 55 and 80 so if by
statistically significant you man 2 Stdev, then with the mean you have.
You would declare every optimization of less than 6% to be statistically
insignificant.
So by what you say here, it seems to me you would have to suggest that
every optimization which provides 6% or less overall speedup to be
removed.
That i doubt many will agree with

Problem 2
We do not meassure speed this way because its not realiable nor practical
just look at this, especially the difference and variation
./ffmpeg -threads 1 -i ~/videos/matrixbench_mpeg2.mpg -f null -
   8941 decicycles in non-intra, 2097003 runs,    149 skipste=N/A speed=53.2x    
   8941 decicycles in non-intra, 2097013 runs,    139 skipste=N/A speed=54.1x    
   8942 decicycles in non-intra, 2097038 runs,    114 skipste=N/A speed=54.1x    
   8970 decicycles in non-intra, 2097037 runs,    115 skipste=N/A speed=  54x    

./ffmpeg -threads 1 -flags2 fast -i ~/videos/matrixbench_mpeg2.mpg -f null -
   8718 decicycles in non-intra, 2097020 runs,    132 skipste=N/A speed=54.6x    
   8701 decicycles in non-intra, 2097044 runs,    108 skipste=N/A speed=54.6x    
   8718 decicycles in non-intra, 2097034 runs,    118 skipste=N/A speed=54.5x    
   8702 decicycles in non-intra, 2097029 runs,    123 skipste=N/A speed=54.5x    
  
This difference is statistically significant, i can say this without the need
to check
Tested on a AMD Ryzen 9 3950X but i expect you will see similar on most CPUs


Problem 3
You dont search for useless code, this is not a "i tested 100 optimizations and found these not worth it"
You search for an argument to remove specific pieces of my code.
thats seriously not making sense to me. really not


Problem 4
try H264 using good old time
time ./ffmpeg -thread_type slice -i fate-suite//h264/bbc2.sample.h264 -f null -
real	0m0,252s
real	0m0,254s
real	0m0,254s
real	0m0,255s

time ./ffmpeg -flags2 fast  -thread_type slice -i fate-suite//h264/bbc2.sample.h264 -f null -
real	0m0,217s
real	0m0,220s
real	0m0,218s
real	0m0,217s

Here even with a crude way of meassuring we can see a clear and strong 
difference


Thanks

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Into a blind darkness they enter who follow after the Ignorance,
they as if into a greater darkness enter who devote themselves
to the Knowledge alone. -- Isha Upanishad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20200614/b30fd249/attachment.sig>