Stream FTP download to output

Question

I am trying to stream/pipe a file to the user's browser through HTTP from FTP. That is, I am trying to print the contents of a file on an FTP server.

This is what I have so far:

public function echo_contents() {                    
    $file = fopen('php://output', 'w+');             

    if(!$file) {                                     
        throw new Exception('Unable to open output');
    }                                                

    try {                                            
        $this->ftp->get($this->path, $file);         
    } catch(Exception $e) {                          
        fclose($file);  // wtb finally               

        throw $e;                                    
    }                                                

    fclose($file);                                   
}                                                    

$this->ftp->get looks like this:

public function get($path, $stream) {
    ftp_fget($this->ftp, $stream, $path, FTP_BINARY);  // Line 200
}

With this approach, I am only able to send small files to the user's browser. For larger files, nothing gets printed and I get a fatal error (readable from Apache logs):

PHP Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 15994881 bytes) in /xxx/ftpconnection.php on line 200

I tried replacing php://output with php://stdout without success (nothing seems to be sent to the browser).

How can I efficiently download from FTP while sending that data to the browser at the same time?

Note: I would not like to use file_get_contents('ftp://user:pass@host:port/path/to/file'); or similar.

Answer 1

Found a solution!

Create a socket pair (anonymous pipe?). Use the non-blocking ftp_nb_fget function to write to one end of the pipe, and echo the other end of the pipe.

Tested to be fast (easily 10MB/s on a 100Mbps connection) so there's not much I/O overhead.

Be sure to clear any output buffers. Frameworks commonly buffer your output.

public function echo_contents() {
    /* FTP writes to [0].  Data passed through from [1]. */
    $sockets = stream_socket_pair(STREAM_PF_UNIX, STREAM_SOCK_STREAM, STREAM_IPPROTO_IP);

    if($sockets === FALSE) {
        throw new Exception('Unable to create socket pair');
    }

    stream_set_write_buffer($sockets[0], 0);
    stream_set_timeout($sockets[1], 0);

    try {
        // $this->ftp is an FtpConnection
        $get = $this->ftp->get_non_blocking($this->path, $sockets[0]);

        while(!$get->is_finished()) {
            $contents = stream_get_contents($sockets[1]);

            if($contents !== false) {
                echo $contents;
                flush();
            }

            $get->resume();
        }

        $contents = stream_get_contents($sockets[1]);

        if($contents !== false) {
            echo $contents;
            flush();
        }
    } catch(Exception $e) {
        fclose($sockets[0]);    // wtb finally
        fclose($sockets[1]);

        throw $e;
    }

    fclose($sockets[0]);
    fclose($sockets[1]);
}

// class FtpConnection
public function get_non_blocking($path, $stream) {
    // $this->ftp is the FTP resource returned by ftp_connect
    return new FtpNonBlockingRequest($this->ftp, $path, $stream);
}

/* TODO Error handling. */
class FtpNonBlockingRequest {
    protected $ftp = NULL;
    protected $status = NULL;

    public function __construct($ftp, $path, $stream) {
        $this->ftp = $ftp;

        $this->status = ftp_nb_fget($this->ftp, $stream, $path, FTP_BINARY);
    }

    public function is_finished() {
        return $this->status !== FTP_MOREDATA;
    }

    public function resume() {
        if($this->is_finished()) {
            throw BadMethodCallException('Cannot continue download; already finished');
        }

        $this->status = ftp_nb_continue($this->ftp);
    }
}

Answer 2

(I've never met this problem myself, so that's just a wild guess ; but, maybe... )

Maybe changing the size of the ouput buffer for the "file" you are writing to could help ?

For that, see stream_set_write_buffer.

For instance :

$fp = fopen('php://output', 'w+');
stream_set_write_buffer($fp, 0);

With this, your code should use a non-buffered stream -- this might help...

Answer 3

Sounds like you need to turn off output buffering for that page, otherwise PHP will try to fit it in all memory.

An easy way to do this is something like:

while (ob_end_clean()) {
    ; # do nothing
}

Put that ahead of your call to ->get(), and I think that will resolve your issue.

Answer 4

Try:

@readfile('ftp://username:password@host/path/file'));

I find with a lot of file operations it's worthwhile letting the underlying OS functionality take care of it for you.

Answer 5

a quick search brought up php’s flush.

this article might also be of interest: http://www.net2ftp.org/forums/viewtopic.php?id=3774

Answer 6

I know this is old, but some may still think it's useful.

I've tried your solution on a Windows environment, and it worked almost perfectly:

$conn_id = ftp_connect($host);
ftp_login($conn_id, $user, $pass) or die();

$sockets = stream_socket_pair(STREAM_PF_INET, STREAM_SOCK_STREAM,
        STREAM_IPPROTO_IP) or die();

stream_set_write_buffer($sockets[0], 0);
stream_set_timeout($sockets[1], 0);

set_time_limit(0);
$status = ftp_nb_fget($conn_id, $sockets[0], $filename, FTP_BINARY);

while ($status === FTP_MOREDATA) {
    echo stream_get_contents($sockets[1]);
    flush();
    $status = ftp_nb_continue($conn_id);
}
echo stream_get_contents($sockets[1]);
flush();

fclose($sockets[0]);
fclose($sockets[1]);

I used STREAM_PF_INET instead of STREAM_PF_UNIX because of Windows, and it worked flawlessly... until the last chunk, which was false for no apparent reason, and I couldn't understand why. So the output was missing the last part.

So I decided to use another approach:

$ctx = stream_context_create();
stream_context_set_params($ctx, array('notification' =>
        function($code, $sev, $message, $msgcode, $bytes, $length) {
    switch ($code) {
        case STREAM_NOTIFY_CONNECT:
            // Connection estabilished
            break;
        case STREAM_NOTIFY_FILE_SIZE_IS:
            // Getting file size
            break;
        case STREAM_NOTIFY_PROGRESS:
            // Some bytes were transferred
            break;
        default: break;
    }
}));
@readfile("ftp://$user:$pass@$host/$filename", false, $ctx);

This worked like a charm with PHP 5.4.5. The bad part is that you can't catch the transferred data, only the chunk size.