Neo4j cpu stuck on GC

Suddenly, after working for one month with almost no use of cpu (between 1 to 5%). The neo4j server is stuck 100% cpu on garbage collecting.

I have neo4j-entherprise 2.0.3 (not embedded) running on ubuntu 4 processors server .

this is my neo4j configuration: wrapper:

wrapper.java.additional=-Dorg.neo4j.server.properties=conf/neo4j-server.properties
wrapper.java.additional=-Djava.util.logging.config.file=conf/logging.properties
wrapper.java.additional=-Dlog4j.configuration=file:conf/log4j.properties

#********************************************************************
# JVM Parameters
#********************************************************************

wrapper.java.additional=-XX:+UseConcMarkSweepGC
wrapper.java.additional=-XX:+CMSClassUnloadingEnabled

# Remote JMX monitoring, uncomment and adjust the following lines as needed.
# Also make sure to update the jmx.access and jmx.password files with appropriate permission roles and passwords,
# the shipped configuration contains only a read only role called 'monitor' with password 'Neo4j'.
# For more details, see: http://download.oracle.com/javase/6/docs/technotes/guides/management/agent.html
# On Unix based systems the jmx.password file needs to be owned by the user that will run the server,
# and have permissions set to 0600.
# For details on setting these file permissions on Windows see:
#     http://download.oracle.com/javase/1.5.0/docs/guide/management/security-windows.html
wrapper.java.additional=-Dcom.sun.management.jmxremote.port=3637
wrapper.java.additional=-Dcom.sun.management.jmxremote.authenticate=true
wrapper.java.additional=-Dcom.sun.management.jmxremote.ssl=false
wrapper.java.additional=-Dcom.sun.management.jmxremote.password.file=conf/jmx.password
wrapper.java.additional=-Dcom.sun.management.jmxremote.access.file=conf/jmx.access

# Some systems cannot discover host name automatically, and need this line configured:
#wrapper.java.additional=-Djava.rmi.server.hostname=$THE_NEO4J_SERVER_HOSTNAME

# disable UDC (report data to neo4j..)
wrapper.java.additional=-Dneo4j.ext.udc.disable=true

# Uncomment the following lines to enable garbage collection logging
wrapper.java.additional=-Xloggc:data/log/neo4j-gc.log
wrapper.java.additional=-XX:+PrintGCDetails
wrapper.java.additional=-XX:+PrintGCDateStamps
wrapper.java.additional=-XX:+PrintGCApplicationStoppedTime
#wrapper.java.additional=-XX:+PrintPromotionFailure
#wrapper.java.additional=-XX:+PrintTenuringDistribution

# Uncomment the following lines to enable JVM startup diagnostics
#wrapper.java.additional=-XX:+PrintFlagsFinal
#wrapper.java.additional=-XX:+PrintFlagsInitial

# Java Heap Size: by default the Java heap size is dynamically
# calculated based on available system resources.
# Uncomment these lines to set specific initial and maximum
# heap size in MB.
#wrapper.java.initmemory=512
wrapper.java.maxmemory=3072

#********************************************************************
# Wrapper settings
#********************************************************************
# path is relative to the bin dir
wrapper.pidfile=../data/neo4j-server.pid

#********************************************************************
# Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
#  using this configuration file has been installed as a service.
#  Please uninstall the service before modifying this section.  The
#  service can then be reinstalled.

# Name of the service
wrapper.name=neo4j

defaults values:

# Default values for the low-level graph engine
neostore.nodestore.db.mapped_memory=25M
neostore.relationshipstore.db.mapped_memory=120M
neostore.propertystore.db.mapped_memory=90M
neostore.propertystore.db.strings.mapped_memory=100M
neostore.propertystore.db.arrays.mapped_memory=100M

What can I do?

EDIT:

The store file sizes:

[
  {
    "description": "Information about the sizes of the different parts of the Neo4j graph store",
    "name": "org.neo4j:instance=kernel#0,name=Store file sizes",
    "attributes": [
      {
        "description": "The total disk space used by this Neo4j instance, in bytes.",
        "name": "TotalStoreSize",
        "value": 401188207,
        "isReadable": "true",
        "type": "long",
        "isWriteable": "false ",
        "isIs": "false "
      },
      {
        "description": "The amount of disk space used by the current Neo4j logical log, in bytes.",
        "name": "LogicalLogSize",
        "value": 24957516,
        "isReadable": "true",
        "type": "long",
        "isWriteable": "false ",
        "isIs": "false "
      },
      {
        "description": "The amount of disk space used to store array properties, in bytes.",
        "name": "ArrayStoreSize",
        "value": 128,
        "isReadable": "true",
        "type": "long",
        "isWriteable": "false ",
        "isIs": "false "
      },
      {
        "description": "The amount of disk space used to store nodes, in bytes.",
        "name": "NodeStoreSize",
        "value": 524160,
        "isReadable": "true",
        "type": "long",
        "isWriteable": "false ",
        "isIs": "false "
      },
      {
        "description": "The amount of disk space used to store properties (excluding string values and array values), in bytes.",
        "name": "PropertyStoreSize",
        "value": 145348280,
        "isReadable": "true",
        "type": "long",
        "isWriteable": "false ",
        "isIs": "false "
      },
      {
        "description": "The amount of disk space used to store relationships, in bytes.",
        "name": "RelationshipStoreSize",
        "value": 114126903,
        "isReadable": "true",
        "type": "long",
        "isWriteable": "false ",
        "isIs": "false "
      },
      {
        "description": "The amount of disk space used to store string properties, in bytes.",
        "name": "StringStoreSize",
        "value": 128,
        "isReadable": "true",
        "type": "long",
        "isWriteable": "false ",
        "isIs": "false "
      }
    ],
    "url": "org.neo4j/instance%3Dkernel%230%2Cname%3DStore+file+sizes"
  }
]

回答1:

Assuming you have 16 GB of RAM in the machine.

First thing is to set the neostore.xxx.mapped_memory settings to match the size of your store files. I'm assuming their total is 5 GB -> you have 11 GB left. See http://docs.neo4j.org/chunked/2.0.4/configuration-caches.html for more details.

Reserve some RAM for the system: 1GB -> you have 10 GB left.

Assign the remaining RAM to java heap using wrapper.java.initmemory wrapper.java.maxmemory. Set both to the same value.

If hpc is used as cache_type consider tweaking its settings based on cache hit ratio for relationships and nodes. Use JMX to monitor them, http://docs.neo4j.org/chunked/2.0.4/jmx-mxbeans.html#jmx-cache-nodecache.

回答2:

We also experienced these kind of issues. In addition to configuration changes similar to what @stefan-armbruster mentioned updating Neo4j to 2.1.2 we also configured Neo4j to use G1 garbage collection instead of CMS.

Since making the garbage collection change we have seen far fewer spikes than we did previously.

If you want to give it a shot you can enable G1 GC by adding the following to your conf/neo4j-wrapper.conf file.