Ruby GC execution exceeding ~250-320ms per request

Question

I have a ruby on rails application. I am investigating an Apdex decline in my NewRelic portal and I'm seeing that on average, 250-320ms of time is being spent on GC execution. This is a highly disturbing number. I've included a screen shot below.

My Ruby version is:

ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]

Any suggestions for tuning this would be ideal. This number should be substantially lower.

Answer 1

There's also this little hack, which may work. But bigger applications tend to allocate such a large amount of memory that you kill a worker per request :-/

Answer 2

You're spending so much time in GC because you're running your GC so often. Ruby, by default, sets GC parameters that are appropriate for small scripts, not large apps. Try launching your app with the following environment parameters set:

RUBY_HEAP_MIN_SLOTS=800000
RUBY_FREE_MIN=100000
RUBY_GC_MALLOC_LIMIT=79000000

What this'll do is increase the initial heap allocation size and pad the GC numbers so that it doesn't run quite so often. This may let your app use a bit more RAM, but it should reduce the time spent in GC dramatically. Under the default settings, you're likely running GC multiple times per request; you want to be ideally be running it once every few requests (or even better, between requests with something like Unicorn's OOB::GC).

Those are my GC settings for my app, and you'll want to tweak them up and down as is most appropriate for your app to find the right settings; you're gunning for a middle ground where you aren't running GC so often, and don't have excessive memory usage. This is specific to each app, though, so there's no boilerplate advice I can give on what those exact settings should be. Increasing from the defaults (10k slots, 1.8x growth factor) should have an immediate impact, and you can tweak up and down from there as best fits your current situtation.

There's a full writup of these parameters here and more information here, and while those posts were written for REE 1.8.7, they're applicable to Ruby 1.9.2+ as well.

Those are some rather extreme numbers, so it's possible that you're doing something in your app that is causing you to allocate much more RAM than you should, so I'd encourage you to be suspicious and comb through your app looking for over-eager allocations. The GC environment variables should help triage the situation in any case, though.

Answer 3

You should use an allocation tracer to find out where your code is allocating objects, and how many. I've used memprof in the past with great results... the big drawback is that it only works under Ruby 1.8 (hopefully your code is 1.8.7-compatible).

If you can run your application under Ruby 1.8.7, then install the "memprof" gem, and:

require 'memprof'
GC.disable
Memprof.track { run_test_code_here }

That will print a listing of allocated object counts (grouped both by class and by the source line where the allocation occurred) to standard output.

When you have problems with excessive time being spent in the garbage collector, usually an allocation trace reveals one or two places where your program is allocating tons of objects. It's impossible to say in advance what the solution will be, but often it will involve either:

1. caching results to avoid repeated calculations,
2. using destructive operations where it is safe to do so (like map! instead of map),
3. reusing a temporary object in a loop (resetting its state each time) rather than allocating a new one on each iteration, or
4. avoiding the use of string constants (or array/hash literals) in a loop which is executed many times (a new object will be allocated on each iteration).

Answer 4

Assuming you are not creating unneeded objects by mistake I heard one hack/solution (besides using JRuby) is to force a GC after finishing sending a response. This way you have the large pause but it is not being seen by the consumer of the request. If you have so much garbage you are seeing multiple pauses like this then you may be out of luck.

This trick may or may not work for your needs.

-JRuby guy who talks to people with MRI GC problems :)