Can't provoke Priority Inversion in C++

Question

I'm trying to provoke Priority Inversion on a small C++ program for demonstration purposes but I can't: The low priority thread that holds the mutex is not preempted and keeps running on the critical section. This is what I'm doing:

// let's declare a global mutex
pthread_mutex_t my_mutex;
  ...

int main(int argc, char **argv) {
  ...
  pthread_t normal_thread;
  pthread_t prio_thread;

  pthread_mutexattr_t attr;
  pthread_mutexattr_init (&attr);
  pthread_mutexattr_setprotocol (&attr, PTHREAD_PRIO_NONE);  // ! None !
  pthread_mutex_init(&my_mutex, &attr);

  // create first normal thread (L):
  pthread_create(&normal_thread, NULL, the_locking_start_routine, NULL);

  // just to help the normal thread enter in the critical section
  sleep(2);

  // now will launch:
  // * (M) several CPU intensive SCHED_FIFO threads with priority < 99
  // * (H) one SCHED_FIFO thread that will try to lock the mutex, with priority < 99

  // build Real Time attributes for the Real Time threads:
  pthread_attr_t my_rt_att;
  pthread_attr_init(&my_rt_att);

  // it was missing in the original post and it was also wrong:
  // even setting the SchedPolicy you have to set "InheritSched"
  pthread_attr_setinheritsched(&my_rt_att, PTHREAD_EXPLICIT_SCHED)

  pthread_attr_setschedpolicy(&my_rt_att, SCHED_FIFO);
  struct sched_param params;

  params.sched_priority = 1;
  pthread_attr_setschedparam(&my_rt_att, &params);

  pthread_create(&prio_thread, &my_rt_att, the_CPU_intensive_start_routine, NULL) 

  params.sched_priority = 99;
  pthread_attr_setschedparam(&my_rt_att, &params);

  // create one RealTime thread like this:
  pthread_create(&prio_thread, &my_rt_att, the_locking_start_routine, NULL)  //coma was missing

  ...
}

void *the_locking_start_routine(void *arg) {
  ...
  pthread_mutex_lock(&my_mutex);
  // This thread is on the critical section
  // ... (skipped)
  pthread_mutex_unlock(&my_mutex);
  ...
}

... But it doesn't work, I can't have my desired Priority Inversion.

This is what happens:

As I understand, with a scheduller like Linux's CFS, a non-real time thread (SCHED_OTHER) will not run until there isn't any Real Time thread (SCHED_FIFO or SCHED_RR) in runnning state. But I have achieved this threads running simultaneously:

• (L) One non-real time (SCHED_OTHER) thread locking the mutex and consuming CPU
• (M) several Real Time threads (SCHED_FIFO , & priority > 0) CPU intensive and non-waiting to lock the mutex
• (H) One Real Time thread (SCHED_FIFO , & highest priority ) waiting for the lock

There are more Real Time CPU intensive threads (M) running than the amount of CPUs of my system ... but the non-real time thread holding (L) the lock is still consuming CPU and finishes it's work and releases the mutex before the "M" threads finish consuming CPU.

Why isn't the low priority thread preempted, the application dead-locked and I can't get priority inversion?

I'm using g++ 4.5.2 on a Ubuntu Desktop 11.04 with kernel 2.6.38-13.

Answer 1

Re: I'm trying to provoke Priority Inversion on a small C++ program for demonstration purposes but I can't: The low priority thread that holds the mutex is not preempted and keeps running ...

That is the beginning of a priority inversion scenario. A low-priority thread grabs an exclusive resource (e.g. mutex) on which high priority threads then block.

To properly show the consequences of priority inversion, you require, for example, three threads: a low (L), middle (M) and high (H) priority thread.

L locks a mutex, for which H contends. So L is running, H is not. This is bad already: important thread H is waiting for less important thread L to do something.

Now M becomes runnable and is compute-intensive. M doesn't care about the mutex; it is not related to H or L. But M has a higher priority than L and kicks L off the CPU.

So now M continues to execute, preventing L from running. That prevents L from reaching the line of code where it releases the mutex, and that prevents H from getting the mutex.

So a middle priority thread M is running instead of the highest priority thread H.

By blocking L, M is able to block H also: inversion.

See if you can code it up exactly like this.

Answer 2

Most modern schedulers have anti-deadlock safeguards that will change the priority for a time slice or two to prevent priority inversion leading to deadlocks, when detected or as deemed appropriate. Whether linux does with whichever scheduler you are using with it, I don't know for sure. However, do be aware of this, if you aren't already.

Answer 3

1. Are you running the program as root?

2. What are your values of these sysctl parameters? Here are mine from an Ubuntu box. The default is to give real time only 0.95 seconds out of a 1 second slice:

   kernel.sched_rt_period_us = 1000000
   kernel.sched_rt_runtime_us = 950000
   

   This prevents the real-time domain from taking all the CPU. If you want real real time, you have to disable these parameters.

See: http://www.kernel.org/doc/Documentation/scheduler/sched-rt-group.txt

If you set sched_rt_runtime_us to -1, you disable this safety mechanism.