Very large uploads with PHP

Question

I want to allow uploads of very large files into our PHP application (hundred of megs - 8 gigs). There are a couple of problems with this however.

Browser:

• HTML uploads have crappy feedback, we need to either poll for progress (which is a bit silly) or show no feedback at all
• Flash uploader puts entire file into memory before starting the upload

Server:

• PHP forces us to set post_max_size, which could result in an easily exploitable DOS attack. I'd like to not set this setting globally.
• The server also requires some other variables to be there in the POST vars, such as an secret key. We'd like to be able to refuse the request right away, instead of after the entire file is uploaded.

Requirements:

• HTTP is a must.
• I'm flexible with client-side technology, as long as it works in a browser.
• PHP is not a requirement, if there's some other technology that will work well on a linux environment, that's perfectly cool.

Answer 1

Maybe you could use Webdav and Javascript in the browser

AJAX Big file upload, with progress, to WebDAV

http://www.webdavsystem.com/ajax/programming/upload_progress

A simple library

http://debris.demon.nl/projects/davclient.js/doc/README.html

You can then get the JS to redirect the user to a success page. Secret keys and what-not can be handled in a PHP prelude before handing off the JS Client->WebDAV

Answer 2

It's old I know, but maybe someone have this problem nowdays ,too. Now you can do this with only Javascript and, say, PHP. No Flash or Java required on client side.

demo: http://dnduploader.filkor.org/

The idea is to slice the files with Javascript's Blob slice() method...

Answer 3

Python Handler?

Using a Python POST handler instead of PHP. Generate a unique identifier from your PHP app that the client can put in the HTTP headers. With mod_python to reject or accept the large upload before the entire POST body is transmitted.

I think http://www.modpython.org/live/current/doc-html/dir-handlers-hph.html

Allows you to check headers and decline the rest of the POST input. I haven't tried it but might be the right path?

Looking at the source of mod_python, the buffering of the input via read() seems to allow bit-at-a-time evaluation of the HTTP input. Headers are first.

https://svn.apache.org/repos/asf/quetzalcoatl/mod_python/trunk/src/filterobject.c

Answer 4

I know it sucks to add another dependency but in my experience, most websites that are doing something like this are using flash on the client side, and uploading the large file as chunks

adobe as a howto on flash file uploads

I also found this tutorial on codeproject:

Multiple File Upload With Progress Bar Using Flash and ASP.NET

PS - I know you're using PHP and not .net, I figured the important part was the flash ;)

Answer 5

upload_max_filesize can be set on a per-directory basis; the same goes for post_max_size

e.g.:

<Directory /uploadpath/>
  php_value upload_max_filesize 10G
  php_value post_max_size 10G
</IfModule>

Answer 6

Have you looked into using APC to check the progress and total file size. Here is a good blog post about it. It might help.