what is the proper way to do logging in csv file?

Question

i want to log some information of every single request send to a busy http server in a formatted form,use log module would create some thing i don't want to:

[I 131104 15:31:29 Sys:34]

i think of csv format but i don't know how to customize it,and python got csv module,but read the manual

import csv
with open('some.csv', 'w', newline='') as f:
    writer = csv.writer(f)
    writer.writerows(someiterable)

since it would open and close a file each time, i am afraid in this way would slow down the whole server performance, what could i do?

Answer 1

As sloth suggests, you can easily edit the delimiter of the log to a comma, thus producing a CSV file.

Working example:

import logging

# create logger
lgr = logging.getLogger('logger name')
lgr.setLevel(logging.DEBUG) # log all escalated at and above DEBUG
# add a file handler
fh = logging.FileHandler('path_of_your_log.csv')
fh.setLevel(logging.DEBUG) # ensure all messages are logged to file

# create a formatter and set the formatter for the handler.
frmt = logging.Formatter('%(asctime)s,%(name)s,%(levelname)s,%(message)s')
fh.setFormatter(frmt)

# add the Handler to the logger
lgr.addHandler(fh)

# You can now start issuing logging statements in your code
lgr.debug('a debug message')
lgr.info('an info message')
lgr.warn('A Checkout this warning.')
lgr.error('An error writen here.')
lgr.critical('Something very critical happened.')

Answer 2

Just use python's logging module.

You can adjust the output the way you want; take a look at Changing the format of displayed messages:

To change the format which is used to display messages, you need to specify the format you want to use:

import logging
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
logging.debug('This message should appear on the console')
logging.info('So should this')
logging.warning('And this, too')

and Formatters:

Formatter objects configure the final order, structure, and contents of the log message.

You'll find a list of the attribtus you can use here: LogRecord attributes.

---

If you want to produce a valid csv-file, use python's csv module, too.

Here's a simple example:

import logging
import csv
import io

class CsvFormatter(logging.Formatter):
    def __init__(self):
        super().__init__()
        self.output = io.StringIO()
        self.writer = csv.writer(self.output, quoting=csv.QUOTE_ALL)

    def format(self, record):
        self.writer.writerow([record.levelname, record.msg])
        data = self.output.getvalue()
        self.output.truncate(0)
        self.output.seek(0)
        return data.strip()

logging.basicConfig(level=logging.DEBUG)

logger = logging.getLogger(__name__)
logging.root.handlers[0].setFormatter(CsvFormatter())

logger.debug('This message should appear on the console')
logger.info('So should "this", and it\'s using quoting...')
logger.warning('And this, too')

Output:

"DEBUG","This message should appear on the console"
"INFO","So should ""this"", and it's using quoting..."
"WARNING","And this, too"

Answer 3

I would agree that you should use the logging module, but you can't really do it properly with just a format string like some of the other answers show, as they do not address the situation where you log a message that contains a comma.

If you need a solution that will properly escape any special characters in the message (or other fields, I suppose), you would have to write a custom formatter and set it.

logger = logging.getLogger()

formatter = MyCsvFormatter()

handler = logging.FileHandler(filename, "w")
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(level)

You'll obviously have to implement the MyCsvFormatter class, which should inherit from logging.Formatter and override the format() method

class MyCsvFormatter(logging.Formatter):
    def __init__(self):
        fmt = "%(levelname)s,%(message)s" # Set a format that uses commas, like the other answers
        super(MyCsvFormatter, self).__init__(fmt=fmt)

    def format(self, record):
        msg = record.getMessage()
        # convert msg to a csv compatible string using your method of choice
        record.msg = msg
        return super(MyCsvFormatter, self).format(self, record)

Note: I've done something like this before, but haven't tested this particular code sample

As far as doing the actual escaping of the message, here's one possible approach: Python - write data into csv format as string (not file)

Answer 4

I don't think that is the best idea, but it is doable, and quite simple. Manually buffer your log. Store log entries in some place, and write them to file from time to time. If you know that your server will be constantly busy, flush your buffer when it reaches some size. If there may be big gaps in usage, I'd say that new thread (or better process, check yourself why threads suck and slow down apps) with endless (theoretically of course) loop of sleep/flush would be better call. Also, remember to create some kind of hook that will flush buffer when server is interrupted or fails (maybe signals? or just try/except on main function - there are even more ways to do it), so you don't lose unflushed buffer data on unexpected exit.

I repeat - this is not the best idea, it's the first thing that came to my mind. You may want to consult logging implementations from Flask or some other webapp framework (AFAIR Flask has CSV logging too).