Analyze GC logs for Sun Hotspots, JVM 6

Question

I'm trying to analyze GC behaviour for our application (running in Tomcat, under Sun's Hotspots, JVM 1.6).

So far I've Instructed the JVM to emit GC logs to a separate file using...

-Xloggc:gc.log 
-XX:+PrintGCApplicationStoppedTime 
-XX:+PrintGCApplicationConcurrentTime 
-XX:+PrintGC 
-XX:+PrintGCTimeStamps 
-XX:+PrintGCDetails

...and used jstat to output logs using...

jstat -gc -t 29045 5s > jstat.gc

I am seeing interesting information, but haven't found a tool to help me analyze/visualize these logs. I was pointed to GCViewer by this question, but it only parses a few log lines from gc.log and then crashes with an exception. Is there a better or more up-to-date tool for parsing these specific logs, for the specific JVM I'm using?

Answer 1

Here is a $0.00 log scraper good for 1.5 CMS collector which gives you a high level view of GC pauses.

You may need to change positional parameter $7 argument to the timestamp function in order to match your log line syntax (my .out gets "enhanced" by Tanuki Wrapper).

#! /usr/bin/awk -f
# Awk script to parse .out logs and print total of
# stop-the-world GC pause times in ten minute intervals

BEGIN {print "t\timark\tmark\tremark\tfullgc"}

/CMS-initial-mark:/ {
  t=timestamp($7);
  imark[t] += $(NF-1);
}

/\[CMS-concurrent-mark:/ {
  t=timestamp($7);
  split($(NF-1), b, "/");
#  print t" NF="NF" val="b[1];
  mark[t] += b[1];
}

/CMS-remark/ {
  t=timestamp($7);
  remark[t] += $(NF-1);
}

/\[Full GC / {
  t=timestamp($7);
  level=0;
  for (i=1; i<=NF; i++) {
      if ($i ~ /\[/) {
        level++;
      } else if ($i ~ /\]/) {
        level--;
      }
  }
  while (level > 0) {
    getline;
    for (i=1; i<=NF; i++) {
      if ($i ~ /\[/) {
        level++;
      } else if ( $i ~ /\]/ ) {
        level-- ;
      }
    }
  }
  if ( $(NF) ~ /secs\]/ ) {
    full[t] += $(NF-1) ;
  }
}

function timestamp(str) {
  split(str, a, ":");
  return a[1]":"substr(a[2],0,length(a[2])-1)"0";
}

# print out UK+US trading hours

END {
  for (hour = 5; hour <= 16; hour++) {
    for (minute = 0; minute <= 59; minute+=10) {
      t = sprintf("%02d:%02d", hour, minute);
      printf "%s\t%d\t%d\t%d\t%d\n", t, imark[t], mark[t], remark[t], full[t];
    }
  }
}

Answer 2

gcviewer hasn't been updated in a number of years, so it's hit or miss - some gc files will work fine, others get exceptions.

IBM's gc log parser works acceptably, http://www.alphaworks.ibm.com/tech/pmat/faq

Sun has something called GCPortal, but it requires:

• install into a web app server
• install of awk and perl
• install of a RDBMS with a JDBC driver and configuration
• and the real killer, third-party graph/chart software that used to be free and now isn't. Amazing.

gchisto has died, there's no longer anything in the project.

HPJmeter doesn't understand IBM gc files.

Answer 3

The tools available in the IBM Support Assistant do a pretty good job.

Answer 4

I personally use HP JMeter for a lot of GC visualization. it works "ok" on SUN JRE's, and exceptionally well on HP's JRE (go figure).

With Sun HotSpot 1.6 (on non-HP platforms), I use these GC Options to produce the logs for analysis:

-Xloggc:/path/to/vgc/log/location/logfile.vgc  --XX:+PrintHeapAtGC 

Answer 5

Have you looked at jvisualvm? Comes with the latest JDK, and allows you to watch the JVM. Sample output (using the visualGC plugin). Sample output - alt text http://g4u0420c.houston.hp.com/en/5992-4687/img/visualgc_2.png

Answer 6

Try using gchisto (gchisto.dev.java.net). It can understand the GC log output (I am not sure whether it has been updated to work with the G1 GC). You have to get the sources from CVS (you need a dev.java.net account for that) and build this yourself