profiling a method of a class in Python using cPro

Question

I'd like to profile a method of a function in Python, using cProfile. I tried the following:

import cProfile as profile

# Inside the class method...
profile.run("self.myMethod()", "output_file")

But it does not work. How can I call a self.method with "run"?

Answer 1

If your function under profile returns value(s), you need to change the excellent answer from @katrielalex slightly:

...             profile.runctx('val = self.baz()', globals(), locals())
...             print locals()['val']

Answer 2

Use the profilehooks decorator

http://pypi.python.org/pypi/profilehooks

Answer 3

If you want to make a cumulative profiler, meaning to run the function several times in a row and watch the sum of the results.

you can use this cumulative_profiler decorator:

import cProfile, pstats

class _ProfileFunc:
    def __init__(self, func, sort_stats_by):
        self.func =  func
        self.profile_runs = []
        self.sort_stats_by = sort_stats_by

    def __call__(self, *args, **kwargs):
        pr = cProfile.Profile()
        pr.enable()  # this is the profiling section
        retval = self.func(*args, **kwargs)
        pr.disable()
        self.profile_runs.append(pr)
        ps = pstats.Stats(*self.profile_runs).sort_stats(self.sort_stats_by)
        return retval, ps

def cumulative_profiler(amount_of_times, sort_stats_by='time'):
    def real_decorator(function):
        def wrapper(*args, **kwargs):
            nonlocal function, amount_of_times, sort_stats_by  # for python 2.x remove this row

            profiled_func = _ProfileFunc(function, sort_stats_by)
            for i in range(amount_of_times):
                retval, ps = profiled_func(*args, **kwargs)
            ps.print_stats()
            return retval  # returns the results of the function
        return wrapper

    if callable(amount_of_times):  # incase you don't want to specify the amount of times
        func = amount_of_times  # amount_of_times is the function in here
        amount_of_times = 5  # the default amount
        return real_decorator(func)
    return real_decorator

Example

profiling the function baz

import time

@cumulative_profiler
def baz():
    time.sleep(1)
    time.sleep(2)
    return 1

baz()

baz ran 5 times and printed this:

         20 function calls in 15.003 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       10   15.003    1.500   15.003    1.500 {built-in method time.sleep}
        5    0.000    0.000   15.003    3.001 <ipython-input-9-c89afe010372>:3(baz)
        5    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

specifying the amount of times

@cumulative_profiler(3)
def baz():
    ...

Answer 4

EDIT: Sorry, didn't realise that the profile call was in a class method.

run just tries to exec the string you pass it. If self isn't bound to anything in the scope of the profiler you are using, you can't use it in run! Use the runctx method to pass in the local and global variables in the scope of the call to the profiler:

>>> import time
>>> import cProfile as profile
>>> class Foo(object):
...     def bar(self):
...             profile.runctx('self.baz()', globals(), locals())
...
...     def baz(self):
...             time.sleep(1)
...             print 'slept'
...             time.sleep(2)
...
>>> foo = Foo()
>>> foo.bar()
slept
         5 function calls in 2.999 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    2.999    2.999 <stdin>:5(baz)
        1    0.000    0.000    2.999    2.999 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        2    2.999    1.499    2.999    1.499 {time.sleep}

Notice the last line: time.sleep is what's taking up the time.

Answer 5

I wouldn't recommend profiling a single routine, because that implies knowing in advance there's a problem there.

A fundamental aspect of performance problems is they're sneaky. They're not where you think they are, because if they were you would have solved them already.

It's better to run the whole program with a realistic workload and let the profiling technique tell you where the problems are.

Here's an example where profiling finds the problem, and it is not where expected.