PyPy displaying inaccurate benchmark results?

Question

I was working on Project Euler and wondered if I could speed up my solution using PyPy. However, I found results quite disappointing, as it took more time to compute.

d:\projeuler>pypy problem204.py
3462.08630405 mseconds

d:\projeuler>python problem204.py
1823.91602542 mseconds

Since mseconds output were calculated using python's time modules, so I ran it again using builtin benchmark commands.

d:\projeuler>pypy -mtimeit -s "import problem204" "problem204._main()"
10 loops, best of 3: 465 msec per loop

d:\projeuler>python -mtimeit -s "import problem204" "problem204._main()"
10 loops, best of 3: 1.87 sec per loop

PyPy reports that it took about half second to finish running. However, I tried running pypy problem204 several times and outputs were never even close to benchmarked .5 seconds. unlike pypy, python's mtimeit results are consistent with outputs. Is pypy giving me inaccurate benchmarks, or is there some magic I don't understand?

Answer 1

Note that timeit

1. runs the statement several times (10 in your case), and
2. does that several times (3 by default) and gives the minimum of that, for reasons outlined in the documentation.

It depends on your code, but it's entirely possible that the JIT compiler is to blame for this confusing result. The JIT warmup overhead is incurred every time you launched a new pypy process, but only once during the timeit benchmark (because that one runs _main several times in the same process). Moreover, if some part of your code is run so often that it's not compiled when _main runs once, but only when it runs, say, three times, subsequent runs will also be faster, which further removes the best result from the first one (i.e. the one for running pypy problem204.py once).

The timeit result is correct in that it (roughly) matches how fast the code will be in the best case - warmed-up JIT compiler, rarely losing the CPU to other programs, etc. Your problem is that you want to know something different - the time including JIT warmup.