How to profile an Erlang program in terms of memor

Question

I would like to further enhance the efficiency of an existing Erlang program. First I would like to identify bottlenecks and then decide on where to further optimize.

I have tryed fprof, but it only gives information on total and average runtime. I would most like to see a log similar to the output of fprof, but in terms of average and total memory usage with respect to functions and processes.

---

For starters it would be enough to profile a single module, that does not spawn processes, only it's functions would be called. This would already help, for I could separate the program to distinct modules for testing.

---

Typical suspicious points are, where bigger lists are being handled.

Here the usage of ++ has been resolved by lists:reverse([Head|Tail]) like syntax.

I am also considering using ETS tables instead of Lists for cases with more than a few hundred elements.

Thank You in advance!

Answer 1

Doing some advertising for my own sake: I wrote a little erlang gen_server a while ago, that records and logs system statistics, combined with a little perl script that parses them and outputs pretty charts.

I've found it pretty useful for doing memory watching etc. under load, as it allows you to continuously monitor a detailed view of the memory usage, while for instance testing different things.

The erlang part is fairly non-intrusive, a simple gen_server that you can start from anywhere, you can just put it under your supervision tree. You can configure the poll frequency etc, and it will write statistics to a file in a simple json format.

The perl script then runs over it, and aggregates the logs to draw charts. There are base-classes, and if you know a little perl, you can easily write a class to log and chart any custom parameter you want.

The script can be obtained from: https://github.com/Amadiro/erlang-statistics

Sample chart (Erlang node that leaks atoms): Sample Chart http://monoc.mo.funpic.de/ram-usage-vs-time.png

Hope this helps you :)

Answer 2

As a more production-ready solution I can recommend collectd erlang client https://github.com/athoune/erlang-collectd if you really need some sort of total memory consumption graphs.

But for more granular memory information you can use process_info(Pid, [memory]) to get memory information for a specific Pid and shell command i() for list of all processes with memory and runtime information. There is also such utilite as etop.

But there is no such profiler like fprof, but for memory usage.

Refer to http://www.erlang.org/faq/how_do_i.html#id52731 for details.

Answer 3

@brainiac, I'm posting new url for erlang-statistics repo on github: https://github.com/Amadiro/erlang-statistics (found it by search ;-)).

Answer 4

The perfect starting point is the Profiling section from the Erlang Efficiency Guide:

http://www.erlang.org/doc/efficiency_guide/profiling.html