How to reduce the memory usage of Appstats on Goog

Question

This is related to question Java Appengine APPSTATS causing java out of memory error.

Appstats seems to cause a java.lang.OutOfMemoryError on 128MB instances and I wonder whether there are ways to reduce the amount of logging. Is there a way to filter some package names from the stack trace?

GAE API:

 @14ms memcache.Get real=7ms api=0ms
Stack:
  com.google.appengine.tools.appstats.Recorder:290 makeAsyncCall()
  com.google.apphosting.api.ApiProxy:184 makeAsyncCall()
  com.google.apphosting.api.ApiProxy:123 makeAsyncCall()
  com.google.appengine.api.memcache.MemcacheServiceApiHelper:104 makeAsyncCall()
  com.google.appengine.api.memcache.AsyncMemcacheServiceImpl:372 doGetAll()
  com.google.appengine.api.memcache.AsyncMemcacheServiceImpl:333 getIdentifiables()
  com.google.appengine.api.memcache.MemcacheServiceImpl:61 getIdentifiables()

my appplication API:

  com.googlecode.objectify.cache.EntityMemcache:215 getAll()
  com.googlecode.objectify.cache.CachingAsyncDatastoreService:253 get()
  com.googlecode.objectify.cache.CachingDatastoreService:161 get()
  com.googlecode.objectify.cache.CachingDatastoreService:147 get()
  com.googlecode.objectify.cache.CachingDatastoreService:128 get()
  siena.gae.GaePersistenceManager:231 getByKey()
  siena.Model:106 getByKey()
  com.sirtrack.iridium.model.GroupEntity:147 getByKey()
  com.sirtrack.iridium.task.ProjectUpdateTask:49 doPost()

irrelevant:

  javax.servlet.http.HttpServlet:637 service()
  javax.servlet.http.HttpServlet:717 service()
  org.mortbay.jetty.servlet.ServletHolder:511 handle()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1166 doFilter()
  com.google.appengine.tools.appstats.AppstatsFilter:141 doFilter()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
  org.vosao.filter.SiteFilter:106 doFilter()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
  org.vosao.filter.RewriteFilter:79 doFilter()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
  org.vosao.filter.AuthenticationFilter:83 doFilter()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
  org.vosao.filter.PluginCronFilter:78 doFilter()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
  org.vosao.filter.LanguageFilter:66 doFilter()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
  org.vosao.filter.UpdateFilter:78 doFilter()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
  org.vosao.filter.InitFilter:80 doFilter()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
  org.vosao.filter.ContextFilter:74 doFilter()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
  com.google.apphosting.utils.servlet.ParseBlobUploadFilter:102 doFilter()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
  com.google.apphosting.runtime.jetty.SaveSessionFilter:35 doFilter()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
  com.google.apphosting.utils.servlet.TransactionCleanupFilter:43 doFilter()
  org.mortbay.jetty.servlet.ServletHandler$CachedChain:1157 doFilter()
  org.mortbay.jetty.servlet.ServletHandler:388 handle()
  org.mortbay.jetty.security.SecurityHandler:216 handle()
  org.mortbay.jetty.servlet.SessionHandler:182 handle()
  org.mortbay.jetty.handler.ContextHandler:765 handle()
  org.mortbay.jetty.webapp.WebAppContext:418 handle()
  com.google.apphosting.runtime.jetty.AppVersionHandlerMap:249 handle()
  org.mortbay.jetty.handler.HandlerWrapper:152 handle()
  org.mortbay.jetty.Server:326 handle()
  org.mortbay.jetty.HttpConnection:542 handleRequest()
  org.mortbay.jetty.HttpConnection$RequestHandler:923 headerComplete()
  com.google.apphosting.runtime.jetty.RpcRequestParser:76 parseAvailable()
  org.mortbay.jetty.HttpConnection:404 handle()
  com.google.apphosting.runtime.jetty.JettyServletEngineAdapter:135 serviceRequest()
  com.google.apphosting.runtime.JavaRuntime$RequestRunnable:446 run()
  com.google.tracing.TraceContext$TraceContextRunnable:449 runInContext()
  com.google.tracing.TraceContext$TraceContextRunnable$1:455 run()
  com.google.tracing.TraceContext:695 runInContext()
  com.google.tracing.TraceContext$AbstractTraceContextCallback:333 runInInheritedContextNoUnref()
  com.google.tracing.TraceContext$AbstractTraceContextCallback:325 runInInheritedContext()
  com.google.tracing.TraceContext$TraceContextRunnable:453 run()
  com.google.apphosting.runtime.ThreadGroupPool$PoolEntry:251 run()
  java.lang.Thread:679 run()

Here's a stack trace of the maybe-caused-by-stacktrace OutOfMemoryError:

Error for /_ah/queue/projectupdate
java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3057)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:407)
    at java.lang.StringBuffer.append(StringBuffer.java:241)
    at java.io.StringWriter.write(StringWriter.java:112)
    at java.io.PrintWriter.write(PrintWriter.java:429)
    at java.io.PrintWriter.write(PrintWriter.java:446)
    at java.io.PrintWriter.print(PrintWriter.java:576)
    at java.io.PrintWriter.println(PrintWriter.java:712)
    at java.lang.Throwable.printStackTrace(Throwable.java:529)
    at com.google.appengine.tools.appstats.Recorder.createStackTrace(Recorder.java:160)
    at com.google.appengine.tools.appstats.Recorder.initializeIntermediary(Recorder.java:271)
    at com.google.appengine.tools.appstats.Recorder.makeAsyncCall(Recorder.java:290)
    at com.googlecode.objectify.cache.TriggerFutureHook.makeAsyncCall(TriggerFutureHook.java:144)
    at com.google.apphosting.api.ApiProxy.makeAsyncCall(ApiProxy.java:184)

For Google Engineers, here's a link to those errors

Answer 1

I've found an undocumented (AFAIK) parameter called "maxLinesOfStackTrace" in the class com.google.appengine.tools.appstats.Recorder.java.

In web.xml:

<filter>
  <filter-name>appstats</filter-name>
    <filter-class>com.google.appengine.tools.appstats.AppstatsFilter</filter-class>
    <init-param>
      <param-name>maxLinesOfStackTrace</param-name>
      <param-value>16</param-value>
    </init-param>
</filter>

It seems to work locally, I'll let you know whether it also helps avoiding the OutOfMemoryError.