How can I tell where my python script is hanging?

Question

So I'm debugging my python program and have encountered a bug that makes the program hang, as if in an infinite loop. Now, I had a problem with an infinite loop before, but when it hung up I could kill the program and python spat out a helpful exception that told me where the program terminated when I sent it the kill command. Now, however, when the program hangs up and I ctrl-c it, it does not abort but continues running. Is there any tool I can use to locate the hang up? I'm new to profiling but from what I know a profiler can only provide you with information about a program that has successfully completed. Or can you use a profiler to debug such hang ups?

Answer 1

If your program has more than one thread, it could be ignoring ctrl-c because the one thread is wired up to the ctrl-c handler, but the live (runaway?) thread is deaf to it. The GIL (global interpreter lock) in CPython means that normally only one thread can actually be running at any one time. I think I solved my (perhaps) similar problem using this

Answer 2

Haven't used it myself but I've heard that the Eric IDE is good and has a good debugger. That's also the only IDE I know of that has a debugger for Python

Answer 3

Wow ! Seems you added so much code in one go without testing it that you can't say what code was added just before program started to hang... (the most likely cause of problem).

Seriously, you should code by small steps and test each one individually (ideally doing TDD).

For your exact problem of spotting what python code is running and ctrl-c does not work, I will try a raw guess: did you used some except: catching all exceptions indistinctly. If you did so in a loop (and continue loop after managing exception), it's a very likely reason why ctrl-c does not work : it's catched by this exception. Change to except Exception: and it should not be catched any more (there is other possibilities for ctrl+c not working like thread management as another poster suggested, but I believe the above reason is more likely).

exception KeyboardInterrupt

Raised when the user hits the interrupt key (normally Control-C or Delete). 

During execution, a check for interrupts is made regularly. Interrupts typed when a built-in function input() or raw_input() is waiting for input also raise this exception. The exception inherits from BaseException so as to not be accidentally caught by code that catches Exception and thus prevent the interpreter from exiting.

Changed in version 2.5: Changed to inherit from BaseException.

Answer 4

You could also try http://code.activestate.com/recipes/576515-debugging-a-running-python-process-by-interrupting/ . It should work as long as the Python process doesn't have signals masked, which is normally the case even if Ctrl-C doesn't work.

Answer 5

I wrote a module that prints out threads that hang longer that 10 seconds at one place. hanging_threads.py

Here is an example output:

--------------------    Thread 5588     --------------------
  File "C:\python33\lib\threading.py", line 844, in _exitfunc
        t.join()
  File "C:\python33\lib\threading.py", line 743, in join
        self._block.wait()
  File "C:\python33\lib\threading.py", line 184, in wait
        waiter.acquire()

This occurs at the exit of the main thread when you forget to set another thread as daemon.

Answer 6

It's easier to prevent these hang-ups than it is to debug them.

First: for loops are very, very hard to get stuck in a situation where the loop won't terminate. Very hard.

Second: while loops are relatively easy to get stuck in a loop.

The first pass is to check every while loop to see if it must be a while loop. Often you can replace while constructs with for, and you'll correct your problem by rethinking your loop.

If you cannot replace a while loop with for, then you simply have to prove that the expression in the while statement must change every time through the loop. This isn't that hard to prove.

1. Look at all the condition in the loop. Call this T.

2. Look at all the logic branches in the body of the loop. Is there any way to get through the loop without making a change to the condition, T?

   • Yes? That's your bug. That logic path is wrong.

   • No? Excellent, that loop must terminate.