PHP multiprocessing with cronjob using CURL

Question

What I need to do is process multiple requests that are pending in a database, using PHP.

What I'm currently trying to do is: When my cronjob runs. I want to call a file "process_a.php" 10 times, instantly, without waiting for it to finish processing (process_a.php could take a few minutes).

I tried to do this using CURL. But right when my cronjob calls process_a.php, it waits for it to finish processing and return before calling another process_a.php file.

I even tried putting code in process_a.php to close the connection instantly, then continue processing in the background. but the cronjob still waited for it to finish.

I just want the same file being executed 10 times at once, you know, like if 10 different users were to request the index.php page of my website... Any ideas!?

Answer 1

Do you have full cron available, or are you only able to specify php files?

You might be able to use xargs to fork 10 processes with the -P argument

seq `1 10`|xargs -n1 -P10 php /path/to/file.php

Answer 2

Fork into 10 processes:

 <?php
 for($i = 0; $i < 10;$i++){
     $pid = pcntl_fork():
     if ($pid == -1) {
          trigger_error('could not fork');
     } else if (!$pid) {
          //we are the child now
           require 'process_a.php';
     } else {
         // we are the parent, you could do something here if need be.
     }
 }

... but that process_a.php could do anything your website does, so, rather then calling a page, why not do the actual work the page request would result in? And let the webserver just continue to be, you know, a webserver instead of a bloated script-repo.

Answer 3

As told by @Brad curl-multi-exec should be an option.

http://php.net/manual/en/function.curl-multi-exec.php

    <?php
//create the multiple cURL handle
$mh = curl_multi_init();

// create both cURL resources
for($i = 0; $i < 10;$i++){
     $ch[$i] = curl_init();
     curl_setopt($ch[$i], CURLOPT_URL, "http://urhost//path/to/process_a.php");
     curl_setopt($ch[$i], CURLOPT_HEADER, 0);
     curl_multi_add_handle($mh,$ch[$i]);//add the handles
}

$active = null;

//execute the handles
do {
    $mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);

while ($active && $mrc == CURLM_OK) {
    if (curl_multi_select($mh) != -1) {
        do {
            $mrc = curl_multi_exec($mh, $active);
        } while ($mrc == CURLM_CALL_MULTI_PERFORM);
    }
}

//close the handles
for($i = 0; $i < 10;$i++){
     curl_multi_remove_handle($mh, $ch[$i]);
}
curl_multi_close($mh);

?>

I tested this script by calling another script below :

<?php 
print microtime();//Return current Unix timestamp with microseconds 
print '<br>';
?>

and here are the results, each handle differing by microseconds in execution time.

0.27085300 1340214659
0.44853600 1340214659
0.46611800 1340214659
0.48201000 1340214659
0.50209400 1340214659
0.48233900 1340214659
0.52274300 1340214659
0.54757800 1340214659
0.57316900 1340214659
0.59475800 1340214659