Linux pthread mutex and kernel scheduler

Question

With a friend of mine, we disagree on how synchronization is handled at userspace level (in the pthread library).

a. I think that during a pthread_mutex_lock, the thread actively waits. Meaning the linux scheduler rises this thread, let it execute his code, which should looks like:

while (mutex_resource->locked);

Then, another thread is scheduled which potentially free the locked field, etc. So this means that the scheduler waits for the thread to complete its schedule time before switching to the next one, no matter what the thread is doing.

b. My friend thinks that the waiting thread somehow tells the kernel "Hey, I'm asleep, don't wait for me at all". In this case, the kernel would schedule the next thread right away, without waiting for the current thread to complete its schedule time, being aware this thread is sleeping.

From what I see in the code of pthread, it seems there is loop handling the lock. But maybe I missed something.

In embedded systems, it could make sense to prevent the kernel from waiting. So he may be right (but I hope he does not :D).

Thanks!

Answer 1

a. I think that during a pthread_mutex_lock, the thread actively waits.

Yes, glibc's NPTL pthread_mutex_lock have active wait (spinning), BUT the spinning is used only for very short amount of time and only for some types of mutexes. After this amount, pthread_mutex_lock will go to sleep, by calling linux syscall futex with WAIT argument.

Only mutexes with type PTHREAD_MUTEX_ADAPTIVE_NP will spin, and default is PTHREAD_MUTEX_TIMED_NP (normal mutex) without spinning. Check MAX_ADAPTIVE_COUNT in __pthread_mutex_lock sources).

If you want to do infinite spinning (active waiting), use pthread_spin_lock function with pthread_spinlock_t-types locks.

I'll consider the rest of your question as if you are using pthread_spin_lock:

Then, another thread is scheduled which potentially free the locked field, etc. So this means that the scheduler waits for the thread to complete its schedule time before switching to the next one, no matter what the thread is doing.

Yes, if there is contention for CPU cores, the your thread with active spinning may block other thread from execute, even if the other thread is the one who will unlock the mutex (spinlock) which is needed by your thread.

But if there is no contention (no thread oversubscribing), and threads are scheduled on different cores (by coincidence, or by manual setting of cpu affinity with sched_setaffinity or pthread_setaffinity_np), spinning will enable you to proceed faster, then using OS-based futex.

b. My friend thinks that the waiting thread somehow tells the kernel "Hey, I'm asleep, don't wait for me at all". In this case, the kernel would schedule the next thread right away, without waiting for the current thread to complete...

Yes, he is right.

futex is the modern way to say OS that this thread is waiting for some value in memory (for opening some mutex); and in current implementation futex also puts our thread to sleep. It is not needed to wake it to do spinning, if kernel knows when to wake up this thread. How it knows? The lock owner, when doing pthread_mutex_unlock, will check, is there any other threads, sleeping on this mutex. If there is any, lock owner will call futex with FUTEX_WAKE, telling OS to wake some thread, registered as sleeper on this mutex.

There is no need to spin, if thread registers itself as waiter in OS.

Answer 2

Some debuging with gdb for this test program:

#include <pthread.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

pthread_mutex_t x = PTHREAD_MUTEX_INITIALIZER;

void* thr_func(void *arg)
{
  pthread_mutex_lock(&x);
}

int main(int argc, char **argv)
{
  pthread_t thr;

  pthread_mutex_lock(&x);
  pthread_create(&thr, NULL, thr_func, NULL);
  pthread_join(thr,NULL);
  return 0;
}

shows that a call to pthread_mutex_lock on a mutex results in a calling a system call futex with the op parameter set to FUTEX_WAIT (http://man7.org/linux/man-pages/man2/futex.2.html)

And this is description of FUTEX_WAIT:

FUTEX_WAIT

This operation atomically verifies that the futex address uaddr still contains the value val, and sleeps awaiting FUTEX_WAKE on this futex address. If the timeout argument is non-NULL, its contents describe the maximum duration of the wait, which is infinite otherwise. The arguments uaddr2 and val3 are ignored.

So from this description I can say that if a mutex is locked then a thread will sleep and not actively wait. And it will sleep until futex with op equal to FUTEX_WAKE is called.