Project Euler and other coding contests often have a maximum time to run or people boast of how fast their particular solution runs. With python, sometimes the approaches are somewhat kludgey - i.e., adding timing code to __main__
.
What is a good way to profile how long a python program takes to run?
A nice profiling module is the line_profiler (called using the script kernprof.py). It can be downloaded here.
My understanding is that cProfile only gives information about total time spent in each function. So individual lines of code are not timed. This is an issue in scientific computing since often one single line can take a lot of time. Also, as I remember, cProfile didn't catch the time I was spending in say numpy.dot.
@Maxy's comment on this answer helped me out enough that I think it deserves its own answer: I already had cProfile-generated .pstats files and I didn't want to re-run things with pycallgraph, so I used gprof2dot, and got pretty svgs:
and BLAM!
It uses dot (the same thing that pycallgraph uses) so output looks similar. I get the impression that gprof2dot loses less information though:
There's also a statistical profiler called
statprof
. It's a sampling profiler, so it adds minimal overhead to your code and gives line-based (not just function-based) timings. It's more suited to soft real-time applications like games, but may be have less precision than cProfile.The version in pypi is a bit old, so can install it with
pip
by specifying the git repository:You can run it like this:
See also https://stackoverflow.com/a/10333592/320036
It's worth pointing out that using the profiler only works (by default) on the main thread, and you won't get any information from other threads if you use them. This can be a bit of a gotcha as it is completely unmentioned in the profiler documentation.
If you also want to profile threads, you'll want to look at the
threading.setprofile()
function in the docs.You could also create your own
threading.Thread
subclass to do it:and use that
ProfiledThread
class instead of the standard one. It might give you more flexibility, but I'm not sure it's worth it, especially if you are using third-party code which wouldn't use your class.The python wiki is a great page for profiling resources: http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Profiling_Code
as is the python docs: http://docs.python.org/library/profile.html
as shown by Chris Lawlor cProfile is a great tool and can easily be used to print to the screen:
or to file:
PS> If you are using Ubuntu, make sure to install python-profile
If you output to file you can get nice visualizations using the following tools
PyCallGraph : a tool to create call graph images
install:
run:
view:
You can use whatever you like to view the png file, I used gimp
Unfortunately I often get
dot: graph is too large for cairo-renderer bitmaps. Scaling by 0.257079 to fit
which makes my images unusably small. So I generally create svg files:
PS> make sure to install graphviz (which provides the dot program):
Alternative Graphing using gprof2dot via @maxy / @quodlibetor :
pprofile
line_profiler
(already presented here) also inspiredpprofile
, which is described as:It provides line-granularity as
line_profiler
, is pure Python, can be used as a standalone command or a module, and can even generate callgrind-format files that can be easily analyzed with[k|q]cachegrind
.vprof
There is also vprof, a Python package described as: