How can I tell where my python script is hanging?

Question

So I'm debugging my python program and have encountered a bug that makes the program hang, as if in an infinite loop. Now, I had a problem with an infinite loop before, but when it hung up I could kill the program and python spat out a helpful exception that told me where the program terminated when I sent it the kill command. Now, however, when the program hangs up and I ctrl-c it, it does not abort but continues running. Is there any tool I can use to locate the hang up? I'm new to profiling but from what I know a profiler can only provide you with information about a program that has successfully completed. Or can you use a profiler to debug such hang ups?

Answer 1

If your program is a bit too complex to simply trace all the functions, you can try running it and manually attaching a tracer program like lptrace to it. It works a bit like strace– it prints every function call your program makes. Here's how to call it:

python lptrace -p $STUCK_PROGRAM_PID

Note that lptrace requires gdb to run.

Answer 2

Nothing like the good old pdb

import pdb
pdb.run('my_method()',globals(),locals())

Then just hit (n) to go to the next command, (s) to step into. see the docs for the full reference. Follow your program step by step, and you'll probably figure it out fast enough.

Answer 3

Wow! 5 answers already and nobody has suggested the most obvious and simple:

1. Try to find a reproducible test case that causes the hanging behavior.
2. Add logging to your code. This can be as basic as print "**010", print "**020", etc. peppered through major areas.
3. Run code. See where it hangs. Can't understand why? Add more logging. (I.e. if between **020 and **030, go and add **023, **025, **027, etc.)
4. Goto 3.

Answer 4

If your program is too big and complex to be viable for single stepping with pdb or printing every line with the trace module then you could try a trick from my days of 8-bit games programming. From Python 2.5 onwards pdb has the ability to associate code with a breakpoint by using the commands command. You can use this to print a message and continue running:

(Pdb) commands 1
(com) print "*** Breakpoint 1 ***"
(com) continue
(com) end
(Pdb)

This will print a message and carry on running when breakpoint 1 is hit. Define similar commands for a few other breakpoints.

You can use this to do a kind of binary search of your code. Attach breakpoints at key places in the code and run it until it hangs. You can tell from the last message which was the last breakpoint it hit. You can then move the other breakpoints and re-run to narrow down the place in the code where it hangs. Rinse and repeat.

Incidentally on the 8-bit micros (Commodore 64, Spectrum etc) you could poke a value into a registry location to change the colour of the border round the screen. I used to set up a few breakpoints to do this with different colours, so when the program ran it would give a psychedelic rainbow display until it hung, then the border would change to a single colour that told you what the last breakpoint was. You could also get a good feel for the relative performance of different sections of code by the amount of each colour in the rainbow. Sometimes I miss that simplicity in these new fangled "Windows" machines.

Answer 5

i = 0
for t in threading.enumerate():
    if i != 0:# and t.getName() != 'Thread-1':
        print t.getName()
        t._Thread__stop()
    i += 1

Once you know the names of the threads; start re-executing your script and filter them down, not stopping them from being aborted. i=0 conditional prevents the main thread from being aborted.

I suggest going through and naming all your threads; such as: Thread(target=self.log_sequence_status, name='log status')

This code should be placed at the end of the main program that starts up the run-away process

Answer 6

Let's assume that you are running your program as:

python YOURSCRIPT.py

Try running your program as:

python -m trace --trace YOURSCRIPT.py

And have some patience while lots of stuff is printed on the screen. If you have an infinite loop, it will go on for-ever (halting problem). If it gets stuck somewhere, then mostly you are stuck on I/O or it is a deadlock.