linux batch jobs in parallel

Question

I have seven licenses of a particular software. Therefore, I want to start 7 jobs simultaneously. I can do that using '&'. Now, 'wait' command waits till the end of all of those 7 processes to be finished to spawn the next 7. Now, I would like to write the shell script where after I start the first seven, as and when a job gets completed I would like to start another. This is because some of those 7 jobs might take very long while some others get over really quickly. I don't want to waste time waiting for all of them to finish. Is there a way to do this in linux? Could you please help me?

Thanks.

Answer 1

You could check how many are currently running and start more if you have less than 7:

while true; do
    if [ "`ps ax -o comm | grep process-name | wc -l`" -lt 7 ]; then
        process-name &
    fi
    sleep 1
done

Answer 2

I found a fairly good solution using make, which is a part of the standard distributions. See here

Answer 3

GNU parallel is the way to go. It is designed for launching multiples instances of a same command, each with a different argument retrieved either from stdin or an external file.

Let's say your licensed script is called myScript, each instance having the same options --arg1 --arg2 and taking a variable parameter --argVariable for each instance spawned, those parameters being stored in file myParameters :

cat myParameters | parallel -halt 1 --jobs 7 ./myScript --arg1 --argVariable {} --arg2

Explanations :

• -halt 1 tells parallel to halt all jobs if one fails
• --jobs 7 will launch 7 instances of myScript

On a debian-based linux system, you can install parallel using :

sudo apt-get install parallel

As a bonus, if your licenses allow it, you can even tell parallel to launch these 7 instances amongst multiple computers.

Answer 4

Write two scripts. One which restarts a job everytime it is finished and one that starts 7 times the first script.

Like:

script1:

./script2 job1
...
./script2 job7

and

script2:

while(...)
  ./jobX