Why is a thread's status running but it doesn&

Question

Today I found a very strange problem. I ran Redhat Enterprise Linux 6, and the CPU was Intel E31275 (4 cores, 8 threads). I found one kernel thread (I called it as my_thread) didn't work correctly. With "ps" command, I found the status of my_thread was always running:

ps ax
5545 ?        R      3:14 [my_thread]
15774 ttyS0    Ss     0:00 -bash
...

But its running time was always 3:14. Since it ws running, why didn't the total time increase? From the proc file /proc/5545/sched, I found the all statistics including wakeups count (se.nr_wakeups) for this thread was always the same, too.

From /proc/5545/stack, I found this thread called this function and never returned:

interruptible_sleep_on_timeout(&q, 3*HZ);

In theory this function would return every 3 seconds if no other threads woke up the thread. Each time after the function returned, se.nr_wakeups in /proc/5545/sched would be increased by 1. But this never happened after I found the thread had some problems.

Does any one have some ideas? Is it possible that interruptible_sleep_on_timeout() never returns?

Update: I find the problem won't occur if I set CPU affinity for this thread. If I pin it to a dedicated core, then everything is OK. Are there any problems with SMP scheduling?

Update again: After I disalbe hyperthread in BIOS, I have not seen such a problem until now.

Answer 1

First off, R indicates the thread is not in running state but runnable. That is, it does not mean it runs, it means it is in a state the scheduler is allowed to pick it for running. There is a big difference between the two.

In a similar sense interruptible_sleep_on_timeout(&q, 3*HZ); will not run the thread after 3 jiffies, but rather make it available for running after 3 jiffies - and indeed you see it in "ps" as available for running, so possibly the timeout has indeed occurred.

Since you did not say anything about the kernel thread in question I don't even know if it is in your own code or standard kernel code so I cannot really answer in detail.

One possible reason for the situation you described is that some other thread (user or kernel) has higher priority then your thread and so the scheduler never picks it for running. If so, it is not probably a thread running in real time priority (SCHED_FIFO or SCHED_RR).