My App Engine-application is having problem with memory leakage.
I log memory usage along the way to find the issue.
from google.appengine.api.runtime import memory_usage
memory_usage().current()
The function that exceeded "soft private memory limit of 128 MB" is within a deferred task. It should behave the same each time.
I re-run it from the consoles task-queue (backend) and from the frontend via get-request. Both get the exception after the 6th log.
The result differs is a way I can't wrap my head around:
<Frontend-run>
1: 40.3515625
2: 50.3515625
3: 59.71875
4: 63.5234375
5: 72.49609375
6: 75.48046875
<Backend-run>
1: 98.83203125
2: 98.83203125
3: 98.83203125
4: 98.83203125
5: 98.83203125
6: 98.83203125
I have three issues with the result:
- One vs. two thirds of total memory-pool is allocated at the start
- Backend uses twice as much memory (running the same function)
- The backend memory usage doesn't increases with time like the frontend does.
Can anyone make sense of this for me?
Apart from the memory usage you'd expect based on the actual activity related to the requests they handle, the instances also have a variable cross-request memory usage offset, including, for example:
- the language (python) sandbox itself
- the additional python libraries loaded while handling previous requests received so far (for example the backend may load the defered library while the frontent might not)
- leftovers not yet cleaned by the garbage collector (they should eventually go away, but occasional activity peaks may cause exceeding the limit and even instance death (and restart) - you'll notice the death happens when the usage goes significantly above the limit, I saw, for example, >150MB for the 128M limit)
On-demand loading of libraries is a typical method of improving instance startup time. Such technique will lead exactly to what may appear a memory leak, but it doesn't necessarily mean it really is a memory leak.
It's also possible that the 128M is simply not enough for an app (you'd be surprised how much may actually be needed and 128M is not a lot!), upgrading the instance type is the only way to move forward. You may actually try it now and monitor the usage - 6 requests is IMHO not sufficient to establish a pattern - if you upgrade and you see the memory usage eventually levelling off then it's likely that you need the upgrade. If it doesn't level off then it's likely that you actually have a memory leak.
