Python3 Flask upload file in server memory

Question

I'm using Flask in Python3 as a webserver, and am using the upload function of Flask. Uploading a file to the server results in a werkzeug.datastructures.FileStorage object.

One of the functions I need this file in, also needs to be able to open files from path objects, so at the moment, I'm using open(file_to_open). If possible, I would like to avoid writing the uploaded file to a temporary file, just to read it in again. So my question consists of two parts:

1: Would it be possible to "translate" this FileStorage object to a file object?

2: If so, would this also work on the current code (open(file_to_open))?

Answer 1

You can store uploaded files to tmpfs. This way they will still be regular files than can be opened with open().

Answer 2

Incoming file uploads are indeed presented as FileStorage objects. However, this does not necessarily mean that an actual physical file is involved.

When parsing file objects, Werkzeug uses the stream_factory() callable to produce a file object. The default implementation only creates an actual physical file for file sizes of 500kb and over, to avoid eating up memory.

For smaller files an in-memory file object is used instead.

I'd not tamper with this arrangement; as it works right now the issue is handled transparently and your harddisk is only involved when the file uploads would otherwise tax your memory too much.

Rather, I'd alter that function to not require a filename and / or accept a file object.

If your function can only take a path or the contained data as a string, you can see if you need to read the file by introspecting the underlying .stream attribute:

from werkzeug._compat import BytesIO

filename = data = None
if file_upload.filename is None:
    data = file_upload.read()  # in-memory stream, so read it out.
else:
    filename = file_upload.filename