Have 5 scripts running at any given time

Question

I have a bash script (running under CentOS 6.4) that launches 90 different PHP scripts, ie.

#!/bin/bash

php path1/some_job_1.php&
php path2/some_job_2.php&
php path3/some_job_3.php&
php path4/some_job_4.php&
php path5/some_job_5.php

php path6/some_job_6.php&
php path7/some_job_7.php&
php path8/some_job_8.php&
php path9/some_job_9.php&
php path10/some_job_10.php
...

exit 0

In order to avoid overloading my server, I use the ampersand &, it works, but my goal is to always have 5 scripts running at the same time

Is there a way to achieve this?

Answer 1

This question is popped several times, but I could not find a proper answer for it. I think now I found a good solution!

Unfortunately parallel is not the part of the standard distributions, but make is. It has a switch -j to do makes parallel.

man make(1)]: (more info on make's parallel execution)

  -j [jobs], --jobs[=jobs]
        Specifies the number of jobs (commands) to run simultaneously.  If
        there  is  more than one -j option, the last one is effective.  If
        the -j option is given without an argument, make  will  not  limit
        the number of jobs that can run simultaneously.

So with a proper Makefile the problem could be solved.

.PHONY: all $(PHP_DEST)

# Create an array of targets in the form of PHP1 PHP2 ... PHP90
PHP_DEST := $(addprefix PHP, $(shell seq 1 1 90))

# Default target
all: $(PHP_DEST)

# Run the proper script for each target
$(PHP_DEST):
        N=$(subst PHP,,$@); php path$N/some_job_$N.php

It creates 90 of PHP# targets each calls php path#/some_job_#.php. If You run make -j 5 then it will run 5 instance of php parallel. If one finishes it starts the next.

I renamed the Makefile to parallel.mak, I run chmod 700 parallel.mak and I added #!/usr/bin/make -f to the first line. Now it can be called as ./parallel.mak -j 5.

Or even You can use the more sophisticated -l switch:

  -l [load], --load-average[=load]
        Specifies  that  no new jobs (commands) should be started if there
        are others jobs running and the load average is at least  load  (a
        floating-point number).  With no argument, removes a previous load
        limit.

In this case make will decide how many jobs can be launched depending on the system's load.

I tested it with ./parallel.mak -j -l 1.0 and run nicely. It started 4 programs in parallel at first contrary -j without args means run as many process parallel as it can.

Answer 2

Use cron and schedule them at the same time. Or use parallel.