Solving PHP memory leaks

Question

I have a PHP script that runs indefinitely long (infinite main event loop) processing a stream of incoming tweets from Twitter and storing them to MySQL. However, I can't seem to get its memory usage under control. I've found 3 ways to measure memory usage:

• memory_get_usage() - Reports about 4.0 MB
• memory_get_usage(true) - Reports about 7.5 MB
• exec("ps -o rss -p " . getmypid(), $memOutput); - Reports a linearly increasing number that quickly grows into the hundreds of MB in 60 minutes or less, and continues eating up memory until the script is forcibly terminated.

My questions:

1) What's the practical difference between these three measures?

But mainly:

2) What does it mean if the first two are relatively constant, but the 3rd method is wildly out of control like this?

FWIW, I'm using PHP 5.3 with Zend Framework 1.x and a lot of Zend_Db activity. Script runs under the CLI SAPI. Zend_Db_Profiler is not being used. I also have a second infinitely-running script that doesn't use the database at all, and memory usage is constant. So it seems to be database-related, perhaps the MySQL extension my PHP setup is using, or maybe Zend_Db. I've taken great pains within my own code to avoid caching objects carelessly, although I haven't done this with Zend's code itself.

I've tried making my script call gc_enable(), and running gc_collect_cycles() periodically, but this doesn't help.

Any ideas?

Edit I intend to profile this code as soon as I can, but meanwhile I've noticed that even my scripts that don't touch the DB are also leaking memory. But they do so at a much slower rate that only becomes apparent when comparing memory usage over the course of several days.

Answer 1

Well I cannot point you to the exact answer here because you need to do the profiling yourself. From what you're saying it seems to point Zend's DB layer, however you cannot be sure unless you profile this. ;)

On UNIX/Linux (and I hope you are running PHP the right way - the UNIX/Linux way:D) there are some very useful tools for profiling such apps on the system level and checking instantiation and memory consumption within the PHP app. You can use Valgrind to get some info, something like:

valgrind --tool=callgrind --dump-instr=yes -v --instr-atstart=no /usr/sbin/apache2 -X

Note that Valgrind is a suite of tools and here we're using the 'callgrind' tool - it

provides all the information that Cachegrind does, plus extra information about callgraphs

This will create a callgrind.out file or a set of them I cannot remember. Anyway you can now use Kcachegrind to visualize the gathered information:

kcachegrind callgrind.out

you will no see a visualization of calls and the percentage of memory a certain part of your app uses. Something like:

You can also try some of the other tools in the Valgrind suite like Memcheck to see

all reads and writes of memory and calls to malloc/new/free/delete

I first learnt about Valgrind when trying to profile my Linux server. Then I researched a bit and it turned out that it is a very good tool to profile PHP apps... There is a very good talk on this here. I used some of the examples from there. Check it out!

And after you profile your app, please come back with information on what was it, or what do you see and etc. I will be very interesting to analyze this. Hope this helped. ;)

EDIT: I now recall that I missed some stuff. :D You can also try using APD wich is a zend extension and could also provide useful information. I haven't personally used it but there are some good examples on the Internet.

Another option is Xhprof - the Hierarchical Profiler. You can use this to gather different metrics. In the end a combination of these tools should be used. How and why is up to you.