[FFmpeg-devel] MPEG-2 FLAGS2_FAST benchmarks

[Michael Niedermayer]

On Sun, Jun 14, 2020 at 12:47:30AM +0200, Michael Niedermayer wrote:
> On Sat, Jun 13, 2020 at 05:11:05PM +0100, Kieran Kunhya wrote:
[...]
> Problem 2
> We do not meassure speed this way because its not realiable nor practical
> just look at this, especially the difference and variation
> ./ffmpeg -threads 1 -i ~/videos/matrixbench_mpeg2.mpg -f null -
>    8941 decicycles in non-intra, 2097003 runs,    149 skipste=N/A speed=53.2x    
>    8941 decicycles in non-intra, 2097013 runs,    139 skipste=N/A speed=54.1x    
>    8942 decicycles in non-intra, 2097038 runs,    114 skipste=N/A speed=54.1x    
>    8970 decicycles in non-intra, 2097037 runs,    115 skipste=N/A speed=  54x    
> 
> ./ffmpeg -threads 1 -flags2 fast -i ~/videos/matrixbench_mpeg2.mpg -f null -
>    8718 decicycles in non-intra, 2097020 runs,    132 skipste=N/A speed=54.6x    
>    8701 decicycles in non-intra, 2097044 runs,    108 skipste=N/A speed=54.6x    
>    8718 decicycles in non-intra, 2097034 runs,    118 skipste=N/A speed=54.5x    
>    8702 decicycles in non-intra, 2097029 runs,    123 skipste=N/A speed=54.5x    

As it was requested, some semi proper statistical analysis while eating breakfast
Iam sure i have made mistakes heres as i dont regularly do such analysis.

Two sample t-test (Welch), using T distribution (DF=5.2188) (left-tailed) (validation)

1. H0 hypothesis
Since p-value < α, H0 is rejected.
The average of fast's population is considered to be less than the average. of the default's population.
In other words, the average of the default's population is bigger than the fast's population, and the difference is big enough to be statistically significant

2. P-value
p-value equals 3.60652e-7, ( p(x≤T) = 3.60652e-7 ). This means that the chance of type1 error (rejecting a correct H0) is small: 3.607e-7 (0.000036%).
The smaller the p-value the more it supports H1.



Two sample mann-whitney u, using Normal tables (n=4.0000) (left-tailed) (validation)

1. H0 hypothesis
Since p-value < α, H0 is rejected.
The randomly selected value of fast's population is considered to be less than the randomly selected value. of the default's population.

2. P-value
p-value equals 0.0142148, ( p(x≤Z) = 0.0142148 ). This means that the chance of type1 error (rejecting a correct H0) is small: 0.01421 (1.42%).
The smaller the p-value the more it supports H1.


Thanks

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Breaking DRM is a little like attempting to break through a door even
though the window is wide open and the only thing in the house is a bunch
of things you dont want and which you would get tomorrow for free anyway
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20200615/260b839e/attachment.sig>