RavenDB: How can I prevent high RAM utilization?

Question

• My application includes a few queries which return large result sets (although I've capped it with a Take(300) lambda.
• During peak usage times, I've seen Raven.Server.exe consume an unusually large amount of RAM.
  • In fact, during these times, Raven.Server.exe can exhaust my server's available RAM.

How can I avoid this situation?

• After a few Google searches, I can see that other people have encountered this error before me.
• But RavenDB has evolved during the past few years and there are many configuration and code options which can limit the amount of RAM that Raven.Server.exe can consume and disable caching entirely.

Can anyone tell me which options are most applicable for my situation?

---

Here is what I found for server configuration options:

• Link 1
• Link 2
• Link 3

Here is what I found for code options:

• Link 1
• Link 2

---

It's clear to me that this default setting is not being respected:

Raven/MemoryCacheLimitMegabytes

The max size in MB for the internal document cache inside RavenDB server. Default: 50% of the total system memory minus the size of the Esent cache.

On my server, with build 2330, without any custom configurations set, Raven.Server.exe was consuming 95% of the available RAM!

---

EDIT: I was able to reproduce this in a test environment when executing only heavy reads (and no writes).

Answer 1

Jim, Do no assume that this is a cache issue. I would assume it is your indexing. Do you have a map/reduce index with multiple from clauses or a SelectMany?

Also, the best place to handle such issues is the mailing list for ravendb.

Answer 2

@Ayende Rahein knows infinitely more about RavenDB than I do, but here is what worked for me:

1. Take(300) was too much. I needed to change this to Take(128) .
2. With a 3.10 GHz quad core client machine, I was peppering a 2.50 GHz single core server with queries inside of a Parallel.ForEach loop. I needed to specify the degree of parallelism: Parallel.ForEach(objects, new ParallelOptions { MaxDegreeOfParallelism = 3 }, currentObject => { /* My Query */ });
3. I needed to configure the following options on my DocumentStore instance: _store.Conventions.DisableProfiling = true; _store.Conventions.ShouldCacheRequest = url => false; _store.DisableAggressiveCaching();
4. And in cases where I needed to page my query and make multiple batch requests within a single session, I needed the following:
   ravenSession.Advanced.Evict(doc); // for each loaded doc

I hope this helps somebody else!