Determine number of idle threads on remote machine

We are working on a script that will tell us how many threads are available on our groups 8 computers. I know that I can use

cat /proc/cpuinfo | grep processor | wc -l

to get the total number of threads, but we would like to know the number of available threads. By available threads I mean how many are currently not in use. We have multiple people accessing these computers and running jobs that require 1-12 threads at a time and would be nice to have a quick query instead of accessing each computer manually

Accessing the computers should be able to be done with this script, I just need to find an appropriate function to use as FindAvaiableThreads()

ssh user@host <<'ENDSSH'
#commands to run on remote host
FindAvaiableThreads()
ENDSSH

ssh user@host2 << 'ENDSSH'
#commands to run on remote host
FindAvaiableThreads()
ENDSSH

...

ssh user@hostN << 'ENDSSH'
#commands to run on remote host
FindAvaiableThreads()
ENDSSH

End Result

We are running checker.scr which sends a script to remote computer n, the script estimates the amount of available cpus by looking at the cpu usage and subtracting from the total amount of cpus and is corrected for hyper-threading as needed.

I'd also like to note that this implementation is based on use of public/private ssh keys to defer password inputs, however sshpass can be used to pass your password to it; however, I am not allowed to install such programs on these computers.

checker.scr

name = username

ssh $name@remote1 'bash -s' < threadquery.scr
ssh $name@remote2 'bash -s' < threadquery.scr
ssh $name@remote3 'bash -s' < threadquery.scr
ssh $name@remote4 'bash -s' < threadquery.scr
ssh $name@remote5 'bash -s' < threadquery.scr
ssh $name@remote6 'bash -s' < threadquery_hyper.scr
ssh $name@remote7 'bash -s' < threadquery_hyper.scr
ssh $name@remote8 'bash -s' < threadquery_hyper.scr

threadquery.scr

NUMCPUS=`grep ^proc /proc/cpuinfo | wc -l`;
FIRST=`cat /proc/stat | awk '/^cpu / {print $5}'`;
sleep 1;
SECOND=`cat /proc/stat | awk '/^cpu / {print $5}'`;
USED=`echo 2 k 100 $SECOND $FIRST - $NUMCPUS / - p | dc`;
AVA=$(echo "$NUMCPUS-$NUMCPUS*$USED/100" | bc -l);
HOSTVAL=$(hostname)
echo "Estimated avaliable processors on $HOSTVAL";
echo $AVA | awk -F\. '{if(($2/10^length($2)) >= .5) printf("%d\n",$1+1);else printf("%d\n",$1)}';

threadquery_hyper.scr

NUMCPUS=`grep ^proc /proc/cpuinfo | wc -l`;
FIRST=`cat /proc/stat | awk '/^cpu / {print $5}'`;
sleep 1;
SECOND=`cat /proc/stat | awk '/^cpu / {print $5}'`;
USED=`echo 2 k 100 $SECOND $FIRST - $NUMCPUS / - p | dc`;
AVA=$(echo "$NUMCPUS-$NUMCPUS*$USED/100" | bc -l);
HOSTVAL=$(hostname)
THREADCORRECT=$(echo "$AVA/2" | bc -l);
echo "Estimated avaliable processors on $HOSTVAL";
echo $THREADCORRECT | awk -F\. '{if(($2/10^length($2)) >= .5) printf("%d\n",$1+1);else printf("%d\n",$1)}';

From my understanding the information you are looking for can be retrieved from 'TOP'

http://www.cyberciti.biz/tips/how-do-i-find-out-linux-cpu-utilization.html

the information is taken from the 'Tasks' section of 'TOP'

#!/bin/bash

getthreads=`top -n 1 | awk '/running/ {print $2, $4, $6}'`
totalthreads=`echo "$getthreads" | awk ' {print $1 } '`
runningthreads=`echo "$getthreads" | awk ' {print $2 } '`
availablethreads=`echo "$getthreads" | awk ' {print $3 } '`

echo "Total threads: $totalthreads"
echo "Threads is use: $runningthreads"
echo "Threads available: $availablethreads"

here is a script that extracts the data you need and outputs it. You can adjust as you need. Hope this helps