How can you profile a Python script?

Question

Project Euler and other coding contests often have a maximum time to run or people boast of how fast their particular solution runs. With python, sometimes the approaches are somewhat kludgey - i.e., adding timing code to __main__.

What is a good way to profile how long a python program takes to run?

Answer 1

I ran into a handy tool called SnakeViz when researching this topic. SnakeViz is a web-based profiling visualization tool. It is very easy to install and use. The usual way I use it is to generate a stat file with %prun and then do analysis in SnakeViz.

The main viz technique used is Sunburst chart as shown below, in which the hierarchy of function calls is arranged as layers of arcs and time info encoded in their angular widths.

The best thing is you can interact with the chart. For example, to zoom in one can click on an arc, and the arc and its descendants will be enlarged as a new sunburst to display more details.

Answer 2

In Virtaal's source there's a very useful class and decorator that can make profiling (even for specific methods/functions) very easy. The output can then be viewed very comfortably in KCacheGrind.

Answer 3

A new tool to handle profiling in Python is PyVmMonitor: http://www.pyvmmonitor.com/

It has some unique features such as

• Attach profiler to a running (CPython) program
• On demand profiling with Yappi integration
• Profile on a different machine
• Multiple processes support (multiprocessing, django...)
• Live sampling/CPU view (with time range selection)
• Deterministic profiling through cProfile/profile integration
• Analyze existing PStats results
• Open DOT files
• Programatic API access
• Group samples by method or line
• PyDev integration
• PyCharm integration

Note: it's commercial, but free for open source.

Answer 4

I think that cProfile is great for profiling, while kcachegrind is great for visualizing the results. The pyprof2calltree in between handles the file conversion.

python -m cProfile -o script.profile script.py
pyprof2calltree -i script.profile -o script.calltree
kcachegrind script.calltree

To install the required tools (on Ubuntu, at least):

apt-get install kcachegrind
pip install pyprof2calltree

The result:

Answer 5

Following Joe Shaw's answer about multi-threaded code not to work as expected, I figured that the runcall method in cProfile is merely doing self.enable() and self.disable() calls around the profiled function call, so you can simply do that yourself and have whatever code you want in-between with minimal interference with existing code.

Answer 6

A while ago I made pycallgraph which generates a visualisation from your Python code. Edit: I've updated the example to work with the latest release.

After a pip install pycallgraph and installing GraphViz you can run it from the command line:

pycallgraph graphviz -- ./mypythonscript.py

Or, you can profile particular parts of your code:

from pycallgraph import PyCallGraph
from pycallgraph.output import GraphvizOutput

with PyCallGraph(output=GraphvizOutput()):
    code_to_profile()

Either of these will generate a pycallgraph.png file similar to the image below: