c++ pthreads - crash while trying to lock mutex fo

Question

Class LocalT have member of other class that realized read-write-mutex. Mutex initialized at constructor and use pthread_rwlock_rdlock(&aMutex); for reading lock. So, seems, its all ok with mutex class. But program crashed when some LocalT object lock his mutex member for reading.

CSerialize.cpp:2054 line is MUTEX.lock_reading();

Thread 6 (Thread 0x80d4e00 (runnable)):
#0  0x4864f11d in pthread_mutex_lock () from /lib/libpthread.so.2
#1  0x4864b558 in pthread_rwlock_init () from /lib/libpthread.so.2
#2  0x4864b659 in pthread_rwlock_rdlock () from /lib/libpthread.so.2
#3  0x0807ae14 in LocalT::serialize (this=0x80d4e00, outbin=@0x7574736b)
    at CSerialize.cpp:2054

Other two running threads:

1) at socket accept();

2) next runnable thread at popen() call, seems its execute or read from pipe. But does not know what is __error() ?????

Thread 1 (Thread 0x8614800 (LWP 100343)):
#0  0x4865b8f9 in __error () from /lib/libpthread.so.2
#1  0x4865a15a in pthread_testcancel () from /lib/libpthread.so.2
#2  0x486425bf in read () from /lib/libpthread.so.2
#3  0x08056340 in UT::execute_popen (command=@0x4865e6bc,
    ptr_output=0xbf2f7d30) at Utils.cpp:75

3) all other thread sleeping.

I have no ideas why its crashed? Maybe somebody can assume something or suggest?

==EDIT==

and here is one system(?) thread ( i does not create it, but program always have +1 thread). It always:

Thread 8 (Thread 0x80d4a00 (LWP 100051)):
#0  0x4865a79b in pthread_testcancel () from /lib/libpthread.so.2
#1  0x48652412 in pthread_mutexattr_init () from /lib/libpthread.so.2
#2  0x489fd450 in ?? ()

==EDIT2 - bt as requested==

(gdb) bt
#0  0x4865a7db in pthread_testcancel () from /lib/libpthread.so.2
#1  0x48652412 in pthread_mutexattr_init () from /lib/libpthread.so.2
#2  0x489fd450 in ?? ()

strangely... why ?? () ?

==EDIT3 - when loading core==

Program terminated with signal 11, Segmentation fault.
[skiped]
#0  0x4865a7db in pthread_testcancel () from /lib/libpthread.so.2
[New Thread 0x8614800 (LWP 100343)]
[New Thread 0x8614600 (sleeping)]
[New Thread 0x8614400 (sleeping)]
[New Thread 0x8614200 (sleeping)]
[New Thread 0x8614000 (sleeping)]
[New Thread 0x80d4e00 (runnable)]
[New Thread 0x80d4c00 (sleeping)]
[New Thread 0x80d4a00 (LWP 100051)]
[New Thread 0x80d4000 (runnable)]
[New LWP 100802]

(gdb) info thread
* 10 LWP 100802  0x4865a7db in pthread_testcancel () from /lib/libpthread.so.2
  9 Thread 0x80d4000 (runnable)  0x486d7bd3 in accept () from /lib/libc.so.6 -- MAIN() THREAD
  8 Thread 0x80d4a00 (LWP 100051)  0x4865a79b in pthread_testcancel ()
   from /lib/libpthread.so.2 ( UNIDENTIFIED THREAD system()? ) 
  7 Thread 0x80d4c00 (sleeping)  0x48651cb6 in pthread_mutexattr_init ()
   from /lib/libpthread.so.2 (SIGNAL PROCESSOR THREAD)
  6 Thread 0x80d4e00 (runnable)  0x4864f11d in pthread_mutex_lock ()
   from /lib/libpthread.so.2 (MAINTENANCE THREAD)
  5 Thread 0x8614000 (sleeping)  0x48651cb6 in pthread_mutexattr_init ()
   from /lib/libpthread.so.2 (other mutex cond_wait - worker 1)
  4 Thread 0x8614200 (sleeping)  0x48651cb6 in pthread_mutexattr_init ()
   from /lib/libpthread.so.2 (other mutex cond_wait - worker 2 )
  3 Thread 0x8614400 (sleeping)  0x48651cb6 in pthread_mutexattr_init ()
   from /lib/libpthread.so.2 (other mutex cond_wait - worker 3 )
  2 Thread 0x8614600 (sleeping)  0x48651cb6 in pthread_mutexattr_init ()
   from /lib/libpthread.so.2 (other mutex cond_wait - worker 4)
  1 Thread 0x8614800 (LWP 100343)  0x4865b8f9 in __error ()
   from /lib/libpthread.so.2 ( popen() thread see below)

I created: 1 maintenance thread (serializing), 1 popen() thread, 4 workers, 1 main, 1 signal thread = 8 threads....

Answer 1

the thread that you are referring to as system thread is actually your program's main thread.

Secondly with information shared by you so far, it looks like you are acquiring the mutex but never releasing it. that leads to an unstable state (some parameters having wrong values) which leads to a crash. I am sure you will also be observing an intermittent hang.

could you share the backtrace when it crashes ?