We're benchmarking some Haskell code in our company and we've just hit a very strange case. Here is a code, which benchmarks the same thing 2 times. The former one uses an Criterion.env
which is created for all the tests once, the later creates env for every test. This is the only difference, however the one which creates env for each bench, runs 5 times faster.
Does anyone know what can cause it? Minimal example:
module Main where
import Prelude
import Control.Monad
import qualified Data.Vector.Storable.Mutable as Vector
import qualified Data.Vector.Storable as Vector
import Data.Vector.Storable (Vector)
import Criterion.Main
testf :: Int -> Vector Int -> IO (Vector Int)
testf !i !v = do
mv <- Vector.unsafeThaw v
let go j = do
x <- if j == 0 then return 0 else Vector.unsafeRead mv (j - 1)
Vector.unsafeWrite mv j (x+1)
when (j < i - 1) $ go (j + 1)
go 0
Vector.unsafeFreeze mv
mkVec :: Int -> IO (Vector Int)
mkVec !i = Vector.unsafeFreeze =<< Vector.new (i + 1)
main :: IO ()
main = do
defaultMain
[ bgroup "env per all runs"
$ (\(i :: Int) -> env (mkVec (10 ^ i))
$ \v -> bench ("10e" ++ show i)
$ nfIO (testf (10 ^ i) v)) <$> [7..8]
, bgroup "env per every run"
$ (\(i :: Int) -> bench ("10e" ++ show i)
$ perRunEnv (mkVec (10 ^ i))
$ (testf (10 ^ i))) <$> [7..8]
]
And results:
benchmarking env per all runs/10e7
time 17.34 ms (17.20 ms .. 17.41 ms)
0.999 R² (0.998 R² .. 1.000 R²)
mean 17.43 ms (17.34 ms .. 17.67 ms)
std dev 321.5 μs (142.1 μs .. 591.3 μs)
benchmarking env per all runs/10e8
time 173.5 ms (173.2 ms .. 173.8 ms)
1.000 R² (1.000 R² .. 1.000 R²)
mean 173.8 ms (173.6 ms .. 174.0 ms)
std dev 279.5 μs (194.9 μs .. 355.6 μs)
variance introduced by outliers: 12% (moderately inflated)
benchmarking env per every run/10e7
time 4.289 ms (1.807 ms .. 5.771 ms)
0.924 R² (0.696 R² .. 1.000 R²)
mean 8.903 ms (5.752 ms .. 14.20 ms)
std dev 5.029 ms (249.0 μs .. 6.244 ms)
variance introduced by outliers: 79% (severely inflated)
benchmarking env per every run/10e8
time 53.76 ms (30.23 ms .. 98.51 ms)
0.940 R² (0.920 R² .. 1.000 R²)
mean 102.9 ms (68.67 ms .. 127.1 ms)
std dev 36.55 ms (0.0 s .. 41.99 ms)
variance introduced by outliers: 73% (severely inflated)