How do I interpret the output of `cargo bench`?

Question

I benchmarked my Rust project with cargo bench and see many numbers on the results... What do they mean?

2 tests
test bench_few_core ... bench:  26,249,920 ns/iter (+/- 2,836,381)
test bench_one_core ... bench:   6,087,923 ns/iter (+/- 752,064)

For example for test bench_few_core, I see:

• number 1 = 26
• number 2 = 249
• number 3 = 920
• number 4 = 2
• number 5 = 836
• number 6 = 381

What do they all mean?

I thought there should be 2 numbers per test: math expectation (or mean) and standard deviation.

Answer 1

Your example does show the two numbers you expect per test: the median and total deviation (i.e. max-min) in nanoseconds per iteration.

Note that for large numbers, it is standard practice in US English to write digits in groups of 3 separated by commas. For example, 26249920 is often written 26,249,920.

Answer 2

The numbers are the average and the difference between the maximum and minimum, expressed using US-centric number styles (which use the comma as the thousands separator).

For your example:

• average: 26249920 ns/iter
• max-min: 2836381 ns/iter

let median = bs.ns_iter_summ.median as usize;
let deviation = (bs.ns_iter_summ.max - bs.ns_iter_summ.min) as usize;

output.write_fmt(format_args!("{:>11} ns/iter (+/- {})",
                              fmt_thousands_sep(median, ','),
                              fmt_thousands_sep(deviation, ',')))

source code

Note that there's various statistical work underlying the benchmarking, most obviously the fact that the upper and lower 5% of samples are truncated to reduce the effect of outliers.