if connection is keep alive how to read until end

Question

$f = fsockopen("www....",80,$x,$y);

fwrite("GET request HTTP/1.1\r\nConnection: keep-alive\r\n\r\n");

while($s = fread($f,1024)){
    ...
}

The above stalls because of the Connection: keep-alive, and works with Connection: close.

How do you do it without stalling?

Answer 1

The following code works without any problem for me:

<?php
$f = fsockopen("www.google.de",80);
fwrite($f,"GET / HTTP/1.1\r\n Connection: keep-alive\r\n\r\n");

while($s = fread($f,1024)){
    echo "got: $s";
}
echo "finished;";
?>

The funny thing is that without keep-alive this example stalls for me. Can you add an example that can be just copy&pasted and shows your error?

Answer 2

It depends on the response, if the transfer-encoding of the response is chunked, then you read until you encounter the "last chunk" (\r\n0\r\n).

If the content-encoding is gzip, then you look at the content-length response header and read that much data and then inflate it. If the transfer-encoding is also set to chunked, then you must dechunk the decoded response.

The easiest thing is to build a simple state machine to read the response from the socket while there is still data left for the response.

When reading chunked data, you should read the first chunk length (and any chunked extension) and then read as much data as the chunk size, and do so until the last chunk.

Put another way:

• Read the HTTP response headers (read small chunks of data until you encounter \r\n\r\n)
• Parse the response headers into an array
• If the transfer-encoding is chunked, read and dechunk the data piece by piece.
• If the content-length header is set, you can read that much data from the socket
• If the content-encoding is gzip, decompress the read data

Once you have performed the above steps, you should have read the entire response and you can now send another HTTP request on the same socket and repeat the process.

On the other hand, unless you have the absolute need for a keep-alive connection, just set Connection: close in the request and you can safely read while (!feof($f)).

I don't have any PHP code for reading and parsing HTTP responses at the moment (I just use cURL) but if you'd like to see actual code, let me know and I can work something up. I could also refer you to some C# code I've made that does all of the above.

EDIT: Here is working code that uses fsockopen to issue an HTTP request and demonstrate reading keep-alive connections with the possibility of chunked encoding and gzip compression. Tested, but not tortured - use at your own risk!!!

<?php

/**
 * PHP HTTP request demo
 * Makes HTTP requests using PHP and fsockopen
 * Supports chunked transfer encoding, gzip compression, and keep-alive
 *
 * @author drew010 <http://stackoverflow.com/questions/11125463/if-connection-is-keep-alive-how-to-read-until-end-of-stream-php/11812536#11812536>
 * @date 2012-08-05
 * Public domain
 *
 */

error_reporting(E_ALL);
ini_set('display_errors', 1);

$host = 'www.kernel.org';

$sock = fsockopen($host, 80, $errno, $errstr, 30);

if (!$sock) {
    die("Connection failed.  $errno: $errstr\n");
}

request($sock, $host, 'GET', '/');

$headers = readResponseHeaders($sock, $resp, $msg);
$body    = readResponseBody($sock, $headers);

echo "Response status: $resp - $msg\n\n";

echo '<pre>' . var_export($headers, true) . '</pre>';
echo "\n\n";
echo $body;

// if the connection is keep-alive, you can make another request here
// as demonstrated below

request($sock, $host, 'GET', '/kernel.css');
$headers = readResponseHeaders($sock, $resp, $msg);
$body    = readResponseBody($sock, $headers);

echo "Response status: $resp - $msg\n\n";

echo '<pre>' . var_export($headers, true) . '</pre>';
echo "\n\n";
echo $body;


exit;

function request($sock, $host, $method = 'GET', $uri = '/', $params = null)
{
    $method = strtoupper($method);
    if ($method != 'GET' && $method != 'POST') $method = 'GET';

    $request = "$method $uri HTTP/1.1\r\n"
              ."Host: $host\r\n"
              ."Connection: keep-alive\r\n"
              ."Accept-encoding: gzip, deflate\r\n"
              ."\r\n";

    fwrite($sock, $request);
}

function readResponseHeaders($sock, &$response_code, &$response_status)
{
    $headers = '';
    $read    = 0;

    while (true) {
        $headers .= fread($sock, 1);
        $read    += 1;

        if ($read >= 4 && $headers[$read - 1] == "\n" && substr($headers, -4) == "\r\n\r\n") {
            break;
        }
    }

    $headers = parseHeaders($headers, $resp, $msg);

    $response_code   = $resp;
    $response_status = $msg;

    return $headers;
}

function readResponseBody($sock, array $headers)
{
    $responseIsChunked = (isset($headers['transfer-encoding']) && stripos($headers['transfer-encoding'], 'chunked') !== false);
    $contentLength     = (isset($headers['content-length'])) ? $headers['content-length'] : -1;
    $isGzip            = (isset($headers['content-encoding']) && $headers['content-encoding'] == 'gzip') ? true : false;
    $close             = (isset($headers['connection']) && stripos($headers['connection'], 'close') !== false) ? true : false;

    $body = '';

    if ($contentLength >= 0) {
        $read = 0;
        do {
            $buf = fread($sock, $contentLength - $read);
            $read += strlen($buf);
            $body .= $buf;
        } while ($read < $contentLength);

    } else if ($responseIsChunked) {
        $body = readChunked($sock);
    } else if ($close) {
        while (!feof($sock)) {
            $body .= fgets($sock, 1024);
        }
    }

    if ($isGzip) {
        $body = gzinflate(substr($body, 10));
    }

    return $body;
}

function readChunked($sock)
{   
    $body = '';

    while (true) {
        $data = '';

        do {
            $data .= fread($sock, 1);
        } while (strpos($data, "\r\n") === false);

        if (strpos($data, ' ') !== false) {
            list($chunksize, $chunkext) = explode(' ', $data, 2);
        } else {
            $chunksize = $data;
            $chunkext  = '';
        }

        $chunksize = (int)base_convert($chunksize, 16, 10);

        if ($chunksize === 0) {
            fread($sock, 2); // read trailing "\r\n"
            return $body;
        } else {
            $data    = '';
            $datalen = 0;
            while ($datalen < $chunksize + 2) {
                $data .= fread($sock, $chunksize - $datalen + 2);
                $datalen = strlen($data);
            }

            $body .= substr($data, 0, -2); // -2 to remove the "\r\n" before the next chunk
        }
    } // while (true)
}

function parseHeaders($headers, &$response_code = null, &$response_message = null)
{
    $lines  = explode("\r\n", $headers);
    $return = array();

    $response = array_shift($lines);

    if (func_num_args() > 1) {
        list($proto, $code, $message) = explode(' ', $response, 3);

        $response_code    = $code;

        if (func_num_args() > 2) {
            $response_message = $message;
        }
    }

    foreach($lines as $header) {
        if (trim($header) == '') continue;
        list($name, $value) = explode(':', $header, 2);

        $return[strtolower(trim($name))] = trim($value);
    }

    return $return;
}