What is going on in this java benchmark?

Question

I'm using junit to run a primitive benchmark like this:

@Test
public void testTime() throws Exception{
    LoadoutBase<?> loadout = new LoadoutStandard("AS7-D-DC");

    final int iterations = 1000000;

    long start = System.nanoTime();
    int sum = 0;
    for(int i = 0; i < iterations; ++i){
        Iterator<Item> it = loadout.iterator();
        while( it.hasNext() ){
            sum += it.next().getNumCriticalSlots();
        }
    }
    long end = System.nanoTime();

    long time_it = end - start;

    start = System.nanoTime();
    sum = 0;
    for(int i = 0; i < iterations; ++i){
        for(Item item : loadout.getAllItems()){
            sum += item.getNumCriticalSlots();
        }
    }
    end = System.nanoTime();

    long time_arrays = end - start;

    System.out.println("it: " + time_it + " array: " + time_arrays + " diff: " + (double)time_it/time_arrays);
}

If I set iterations=1000000 then I get

it: 792771504 array: 1109215387 diff: 0.7147137637029551

very consistently but if I set iterations=10000 then I get

it: 32365742 array: 28902811 diff: 1.1198129482976587

with very wild fluctuations. The diff parameters is anywhere from 0.7 to 1.2

Could any one shed some light on what might be happening here? Which method should I choose?

Edit:

What I really am benchmarking is behind the scenes work. getAllItems creates a new List<Item> and populates it by getting all items from 16 sublists using addAll. The Iterator approach doesn't construct this temporary list but rather keeps track of in which of these 16 sublists it is currently iterating and has some logic to make the 16 sublists look like a continuous list.

Answer 1

After reading the links from Luiggi Mendoza, I refactored my test code like this:

@Test
public void testTime() throws Exception{

    LoadoutBase<?> loadout = new LoadoutStandard("AS7-D-DC");

    long start, end, sum;
    final int iterations = 10000;

    //WARMUP
    start = System.nanoTime();
    sum = 0;
    for(int i = 0; i < iterations; ++i){
        Iterator<Item> it = loadout.iterator();
        while( it.hasNext() ){
            sum += it.next().getNumCriticalSlots();
        }
    }
    end = System.nanoTime();
    //ENDOF WARMUP

    start = System.nanoTime();
    sum = 0;
    for(int i = 0; i < iterations; ++i){
        Iterator<Item> it = loadout.iterator();
        while( it.hasNext() ){
            sum += it.next().getNumCriticalSlots();
        }
    }
    end = System.nanoTime();
    long time_it = end - start;


    // WARMUP
    start = System.nanoTime();
    sum = 0;
    for(int i = 0; i < iterations; ++i){
        for(Item item : loadout.getAllItems()){
            sum += item.getNumCriticalSlots();
        }
    }
    end = System.nanoTime();
    // ENDOF WARMUP


    start = System.nanoTime();
    sum = 0;
    for(int i = 0; i < iterations; ++i){
        for(Item item : loadout.getAllItems()){
            sum += item.getNumCriticalSlots();
        }
    }
    end = System.nanoTime();
    long time_arrays = end - start;

    System.out.println("it: " + time_it + " array: " + time_arrays + " diff: " + (double)time_it/time_arrays);
}

Now I get consistent results saying that the iterator approach is about 0.7 times the execution speed of the getAllItems approach. The results are consistent even if I change the order of the tests so I trust them for this test.

Thanks.

Answer 2

Since you want to test the difference between using Iterator and using enhanced for loop (which uses Iterator behind the scenes for you), then you're doing it wrong. First because JIT has enough time to improve the results on the second approach rather than in the first and several other reasons explained here: How do I write a correct micro-benchmark in Java?. You could see very different results of this by doing these (again, as result of JIT):

• Add a loop that will increase the number of times to execute both iterations. This is, a for loop that covers all this code.
• Moving the enhanced for loop approach to be executed before.

The best way to obtain real results for your test would be to split the approaches into different methods, then measure each (apart from the JVM warm up phase and other stuff covered in prev QA). Also, I recommend you to not reinvent the wheel and use a proper benchmark framework based on JUnit. Here are two of them:

• JUnitBenchmarks
• Caliper

Related Q/As for benchmarking:

• arraylist vs linked list .why linked list is slower when we add in the end?
• Decreasing execution time of identical consecutive runs of a Java program