Curl and pcntl_fork()

I have some code that is used to check links on a website and in trying to make it "threaded", the code was updated to use pcntl_fork().

The parent code is working for SSL and non-SSL URL's but the child code is only working for non-SSL URL's. I've noted in the code where it works and where it doesn't.

Here is my fork code. I know the code below will loop forever, I have taken out the loop control code so it is more readable.

$this->initialize_curl();
$this->connect_database();

// prime the queue
$this->add_url_to_queue($this->source_url, 0, 0);
$this->process_next_url_in_queue($this->get_next_url_in_queue());

// SSL and non-SSL work at this point

// loop until we have processed all URL's
while (1) {
  $url = $this->get_next_url_in_queue();

  // disconnect from the database before forking since we don't want to
  // share the database connection with child processes - the first one
  // will close it and ruin the fun for the other children.
  curl_close($this->ch);
  $this->db->close();

  // create child
  $pid = pcntl_fork();

  // handle forked processing
  switch ($pid) {

    // error
    case -1:
      print "Could not fork\n";
      exit;

    // child
    case 0:

      // seperate database and curl for the child
      $this->connect_database();
      $this->initialize_curl();

      // process the url
      $this->process_next_url_in_queue($url);

      // only non-SSL works at this point

      exit;

    // parent
    default:

      // seperate database and curl for the parent
      $this->connect_database();
      $this->initialize_curl();
      break;
  }
}

As you can see, I had to open and close the database connection so it would work and I am doing the same with CURL. Here is the code in initialize_curl():

$this->ch = curl_init();
curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($this->ch, CURLOPT_FOLLOWLOCATION, FALSE);
curl_setopt($this->ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($this->ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($this->ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($this->ch, CURLOPT_HEADER,         FALSE);

I am using CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST because without it my SSL CURL requests will fail. It's an issue with the server set up and not something I can change.

Wwhen a child CURL's an SSL URL, I think it fails because there is a problem with setting these options, but I don't know. If I set CURL to be verbose, I see the following error:

* About to connect() to HOST port 443 (#0)
*   Trying IP... * connected
* Connected to HOST (IP) port 443 (#0)
* NSS error -8023
* Closing connection #0
* SSL connect error

Please let me know what I can do to make this work.

After a lot of research, I uncovered that the issue is not a new and is a problem with php's implementation of CURL. These other questions helped me to come up with the solution I've shared below:

SSL Requests made with cURL fail after process fork
libCurl SSL error after fork()

What I ended up doing was to use pcntl_exec which replaces the current child process with the command provided.

$this->initialize_curl();
$this->connect_database();

// prime the queue
$this->add_url_to_queue($this->source_url, 0, 0);
$this->process_next_url_in_queue($this->get_next_url_in_queue());

// loop until we have processed all URL's
while (1) {
  $url = $this->get_next_url_in_queue();

  // disconnect from the database before forking since we don't want to
  // share the database connection with child processes - the first one
  // will close it and ruin the fun for the other children.
  curl_close($this->ch);
  $this->db->close();

  // create child
  $pid = pcntl_fork();

  // handle forked processing
  switch ($pid) {

    // error
    case -1:
      print "Could not fork\n";
      exit;

    // child
    case 0:

      // seperate database and curl for the child
      $this->connect_database();
      $this->initialize_curl();

      // process the url
      pcntl_exec('process_next_url_in_queue.php', array($url));

      exit;

    // parent
    default:

      // seperate database and curl for the parent
      $this->connect_database();
      $this->initialize_curl();
      break;
  }
}