Debugging deadlock with pthread mutex(linux)

Question

i am facing a deadlock in one of my c application(its a big code) and I was able to debug down the stage where I printed a mutex. It looks like below -

          {__data = 
              {__lock = 2,
               __count = 0, 
               __owner = 15805, 
               __nusers = 1, 
               __kind = 0, 
               __spins = 0, 
               __list = {__prev = 0x0, __next = 0x0}
              },
            __size = "\002\000\000\000\000\000\000\000½=\000\000\001", '\0' <repeats 26 times>, 
            __align = 2
         }

Now i could understand that __owner is thread id of thread holding this mutex, same thread ends up in deadlock for this mutex. Does anyone know meaning of rest of fields such as _lock,_count,__spins etc which could to be useful in debugging deadlocks? Any tips/tricks would also be welcome? (Based on debugging, i understood thread tries to lock mutex which it already holds, ending up in deadlock and also other threads are wating for this mutex)

Also is this possible to find out at which line of code has this been locked by observing process/threads through debugger(gdb) (of course debug info and code is at hand) without requiring careful code-walkthrough of code? I have gone through my code several times but was unable to find where this lock has been left unreleased before returning from function.

Thanks Nerdy

Answer 1

The other fields are not particularly useful for debugging deadlocks. The important information is what you've already found - the thread is deadlocking because it is locking a mutex that is already locked by itself.

To debug this with gdb, you could set breakpoints at each line where that mutex is locked by the thread of interest. Each time the breakpoint triggers, just continue execution. When the deadlock happens, the most recent previous lock operation is obviously the one that has no corresponding unlock.