How to log large requests as files?

Question

I am using a Flask app to solve vehicle routing problems. It receives large requests (around 5MB each) that I would like to log outside of the usual logging messages (e.g. app.logger.info(..)) because they just render the logs unreadable.

My idea is to save the last n requests as json files with timestamps in a separate folder. I am about to write my own function to do this. I am wondering if you have used this type of setup before and how you implemented it. I couldn't find anything in Python's module logging to do this for me.

Answer 1

I think you should ignore the logging module in this case.

Take a directory and create the files in this directory.

You can use the current datetime for the file_name.

It has a paranoid way of creating a unique name:

import datetime
import os
import random

request_dir = '/tmp'
content =':-)'
now = datetime.datetime.now()
file_name = '%s_%s_%s.json' % (now.strftime('%Y-%m-%d_%H%M%S'), '%06d' % now.microsecond,
                          random.randint(100000000, 999999999))
with open(os.path.join(request_dir, file_name), 'wb') as fd:
    fd.write(content)

The part "deleting old entries in the directory" is left undone. This is not difficult since the file names can be sorted.

Answer 2

I wouldn't recommend using the logger module to save large json files, especially if you also want them formatted for human consumption.

I actually had the same problem as you in the past, and solved it writing the following function:

def json_to_file(file_to_write, json_to_dump):
    max_files = 10
    file_list = sorted(glob.glob("your pattern here"))
    if len(file_list) > max_files:
        os.remove(file_list[0])

    with open(file_to_write, "w+") as my_file:
        json.dump(json_to_dump, my_file, indent=4)

You could format the file_to_write variable to be a unix or utc timestamp - that's what I do anyway.

Edit: added the "max n files" part. Just make sure you're using the right pattern with glob!

Answer 3

If you want to use python logger, you can specify where it writes using FileHandler like this:

fh = logging.FileHandler(filename=os.path.join(path_to_log_directory, log_filename))
fh.setLevel(LOG_LEVEL)