Benchmarking : Same process multiple times, only o

Question

I am currently working on a Java application (Benchmark) that has for purpose to mesure some processes relative to a database.

My application is supposed to run has following :

I have several Usecases (simple insert, simple update, etc., in a database) that I would like to run multiple times. The only difference between the runs, would be the number of threads running at the same time.

I need to bench these usecases using 1, 2, 4, 8, 16, etc.. threads in order to include concurrency in my tests (Using ExecutorService).

My question :

Does my app need to run a warmup before each run ? or only one is sufficient. In other words, does my app has to do the following :

--warmup
--process(1) (1 thread)
--warmup
--process(2) (2 threads)
etc.

OR

--warmup
--process(1)
--process(2)
etc.

Basically, the "process()" method is exactly the same, regardless of the number of threads. I tend to believe that one is clearly sufficient, as the JVM will not really optimize anything as the code does not change. But, still, I prefer to seek some exerimented advices :)

Thank you for your help !

Note : I read a lot about benchmarking :

• http://www.ibm.com/developerworks/library/j-jtp12214/
• http://www.ibm.com/developerworks/java/library/j-benchmark1/index.html
• http://www.ibm.com/developerworks/java/library/j-benchmark2/

This is why I would say that only one warmup is necessary. :)

Answer 1

One warmup is needed. If you covered those articles and doing a proper warmup (enough iterations, making sure your warmup code doesn't get eliminated etc.) then your warmup will be sufficient and you won't get any extra benefits of running it prior to each test.