linux worker script/queue (php)

Question

I need a binary/script (php) that does the following.

Start n process of X in the background and maintain the number processes.

An example:

• n = 50
• initially 50 processes are started
• a process exits
• 49 are still running
• so 1 should be started again.

P.S.: I posted the same question on SV, which makes me probably very unpopular.

Answer 1

I would go in the direction that andres suggested. Just put something like this at the top of your pro.php file...

$this_file = __FILE__;
$final_count = 50;

$processes = `ps auwx | grep "php -f $this_file"`;
$processes = explode("\n", $processes);
if (count($processes)>$final_count+3) {
        exit;
}
//... Remaining code goes here

Answer 2

Have you tried making a PHP Daemon before?

http://kevin.vanzonneveld.net/techblog/article/create_daemons_in_php/

Answer 3

Pseudocode:

for (i=1; i<=50; i++)
  myprocess
endfor

while true
  while ( $(ps --no-headers -C myprocess|wc -l) < 50 )
    myprocess
  endwhile
endwhile

If you translate this to php and fix its flaws, it might just do what you want.

Answer 4

Can you use the crontab linux and write to a db or file the number of current process?. If DB, the advantage is that you can use to procedure and lock the table, and write the number of process.

But to backgroun you should use & at the end of the call to script

# php-f pro.php &

Answer 5

Here's something in Perl I have in my library (and hey, let's be honest, I'm not going to rig this up in PHP just to give you something working in that language this moment. I'm just using what I can copy / paste).

#!/usr/bin/perl
use threads;
use Thread::Queue;

my @workers;
my $num_threads = shift;
my $dbname = shift;
my $queue = new Thread::Queue;

for (0..$num_threads-1) {
        $workers[$_] = new threads(\&worker);
                print "TEST!\n";
}

while ($_ = shift @ARGV) {
        $queue->enqueue($_);
}

sub worker() {
        while ($file = $queue->dequeue) {
                system ('./4parser.pl', $dbname, $file);
        }
}

for (0..$num_threads-1) { $queue->enqueue(undef); }
for (0..$num_threads-1) { $workers[$_]->join; }

Whenever one of those systems calls finishes up, it moves on dequeing. Oh, and damn if I know hwy I did 0..$numthreads instead of the normal my $i = 0; $i < ... idiom, but I did it that way that time.

Answer 6

I have to solutions to propose. Both do child process reboot on exit, do child process reloading on USR1 signal, wait for the children exit on SIGTERM and so on.

The first is based on swoole php extension. It is very performant, async, non-blocking. Here's the usage example code:

<?php
use Symfony\Component\Process\PhpExecutableFinder;

require_once __DIR__.'/../vendor/autoload.php';
$phpBin = (new PhpExecutableFinder)->find();
if (false === $phpBin) {
    throw new \LogicException('Php executable could not be found');
}
$daemon = new \App\Infra\Swoole\Daemon();

$daemon->addWorker(1, $phpBin, [__DIR__ . '/console', 'quartz:scheduler', '-vvv']);

$daemon->addWorker(3, $phpBin, [__DIR__ . '/console', 'enqueue:consume', '--setup-broker', '-vvv']);

$daemon->run();

The daemon code is here

Another is based on Symfony process library. It does not require any extra extensions. The usage example and daemon code could be found here