Benchmarking inside Java code

Question

I have been looking into benchmarking lately, I have always been interested in logging program data etc. I was interested in knowing if we can implement our own memory usage code and implement our own time consumption code efficently inside our program. I know how to check long it takes for a code to run:

public static void main(String[]args){
        long start = System.currentTimeMillis();
        // code
        System.out.println(System.currentTimeMillis() - start);
    }

I also looked into Robust Java benchmarking, Part 1: Issues, this tutorial is very comprehensive. Displays the negative effects of System.currentTimeMillis();. The tutorial then suggests that we use System.nanoTime(); (making it more accurate?).

I also looked at Determining Memory Usage in Java for memory usage. The website shows how you can implement it. The code that has been provided looks inefficent because the person is calling

long L = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();

After this he calls System.gc(); (4 * 4) = 16 times. Then repeating the process again. Doesn't this also take up memory?

So in conlusion, is it possible to implement an efficent benchmarking code inside your java program?

Answer 1

Yes it is possible to effectively implement performance benchmarks in java code. The important question is that any kind of performance benchmark is going to add its own overhead and how much of it do you want. System.currentMill..() is good enough benchmark for performance and in most of the cases nanoTime() is an overkill.

For memory System.gc will show you varied results for different runs (as gc run is never guranteed.) I generally use Visual VM for memory profiling (its free) and then use TDA for dumps analyzing.

One way to do it less invasively is using Aspect oriented programing. You can create just one Aspect that runs on a particular Annotation or set of methods and write an @Around advice to collect performance data.

Here is a small snippet:

public class TestAspect {

    @LogPerformance
    public void thisMethodNeedsToBeMonitored(){
        // Do Something
    }
    public void thisMethodNeedsToBeMonitoredToo(){
        // Do Something
    } 
}

@interface LogPerformance{}

@Aspect
class PerformanceAspect{
    @Around("the pointcut expression to pick up all " +
            "the @PerfMonitor annotated methods")
    public void logPerformance(){
        // log performance here
        // Log it to a file
    }
}

Answer 2

It may be impossible to benchmark without some Heisenberg effect, i.e. your benching code also being measured. However, if you measure at a high enough granularity the effect will be negligible.

Answer 3

Any benchmarking code is going to be less efficient than non-benchmarked code just based on having more stuff to do. That said, Java in particular causes issues as the article states due to garbage collection happening whenever the jre feels like it. Even the documentation for System.gc says it makes a "best effort".

As for your specific questions:

System.gc shouldn't take up more memory, but it will take processor resources.

It is somewhat possible based on what you're trying to benchmark. There will always be some interference. If you are willing to go outside of your code, there are tools like VisualVM to watch memory usage from outside of your application.

Edit: Corrected the wording of what System.gc docs say.