Unable to profile JBoss 5 using jvisualvm

I've been getting some java.lang.OutOfMemoryError: GC overhead limit exceeded errors while running my Java app overnight:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid6376.hprof ...
Heap dump file created [512149941 bytes in 23.586 secs]
23:34:52,163 WARN  [HDScanner] Scan failed
java.lang.OutOfMemoryError: Java heap space
23:34:52,298 ERROR [ContainerBase] Exception invoking periodic operation: 
java.lang.OutOfMemoryError: Java heap space
23:34:52,321 ERROR [JIoEndpoint] Socket accept failed
java.lang.OutOfMemoryError: Java heap space
    at java.net.ServerSocket.accept(Unknown Source)
    at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
    at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)

If I open up jvisualvm, I can see that I am indeed out of heap space:

I'd like to profile it to try to figure out what's going on - is there a memory leak? etc. However, I'm unable to get the jvisualvm profiler to do anything at all. The Profiler tab shows a warning about class sharing being enabled:

...even though I've added the -Xshare:off flag to my VM args:

So,

Any ideas about how to get profiling to work?
Is it worth playing around with G1GC instead of ...whatever GC I'm currently using?

It looks to me that you are running VisualVM on 32-bit JVM and your JBoss is running on 64-bit JVM. In such situation profiler and other things like Threads and CPU monitoring does not work on Windows out of the box. With profiler you are running into bug #273, where profiler is incorrectly enabled. You should do the following:

Dowload latest 64-bit JDK (currently JDK 6u23). It also contains the latest version of Java VisualVM
Use memory sampler in VisualVM (it has much lower overhead than profiling) to see if you can spot the problem.
If that does not help, take a look at the heap dump and try to find out what is going on.
Do memory profiling of your JBoss server. See Profiling With VisualVM, Part 1 and Profiling With VisualVM, Part 2 for more information about profiler in VisualVM.