[Ffmpeg-devel] Re: [PATCH] h264 - loopify some get_cabac calls

[Trent Piepho]

On Thu, 19 Apr 2007, Alexander Strange wrote:
> On Apr 19, 2007, at 10:05 PM, Trent Piepho wrote:
>
> > On Thu, 19 Apr 2007, Alexander Strange wrote:
> >> I finally found the time to look through the rest of the benchmarks I
> >> did for this:
> > [...]
> >> faculty.vassar.edu/lowry/wilcoxon.html:
> >> z = 2.78 / p = .0027 (95% significant)
> >> z = 2.47 / p = .0068 (same)
> >>
> >> I don't think this is quite true, since the numbers are running
> >> averages and not individual samples, but without std. deviations it's
> >> the best I can do.
> >
> > Since you just have just one run, you can't do any kind of
> > statistical analysis.  Using a running average at different
> > points as if they were independent observations is completely
> > invalid.
>
> What did you do earlier in the thread?

I must have thought your data was from multiple runs.

> > Why are there so many skips?
>
> It skips once whenever it handles a skip macroblock. Probably it
> should skip START_TIMER as well, but I did it a while ago and the
> numbers should be the same.
>
> I don't like benchmarking this way at all. I wonder if there are any
> x86 simulators as good as simg4/simg5 for PPC.

Modern CPUs and the busses and memory interfaces that support
them are complicated.  The code you want to benchmark doesn't exist
in a vacuum; its run time is affected by the cpu state (and that's
not just the cache, though that's a major part of it) before it
is run, and it effects the run time of the code that follows it.

It would be easier if you could analyze the code and get
some constant cycle count, but that's just not the way it works.