Multiline log records in syslog

Question

So I've configured my Python application to log to syslog with Python's SysLogHandler, and everything works fine. Except for multi-line handling. Not that I need to emit multiline log records so badly (I do a little), but I need to be able to read Python's exceptions. I'm using Ubuntu with rsyslog 4.2.0. This is what I'm getting:

Mar 28 20:11:59 telemachos root: ERROR 'EXCEPTION'#012Traceback (most recent call last):#012  File "./test.py", line 22, in <module>#012    foo()#012  File "./test.py", line 13, in foo#012    bar()#012  File "./test.py", line 16, in bar#012    bla()#012  File "./test.py", line 19, in bla#012    raise Exception("EXCEPTION!")#012Exception: EXCEPTION!

Test code in case you need it:

import logging
from logging.handlers import SysLogHandler

logger = logging.getLogger()
logger.setLevel(logging.INFO)
syslog = SysLogHandler(address='/dev/log', facility='local0')
formatter = logging.Formatter('%(name)s: %(levelname)s %(message)r')
syslog.setFormatter(formatter)
logger.addHandler(syslog)

def foo():
    bar()

def bar():
    bla()

def bla():
    raise Exception("EXCEPTION!")

try:
    foo()
except:
    logger.exception("EXCEPTION")

Answer 1

Another option would be to subclass the SysLogHandler and override emit() - you could then call the superclass emit() for each line in the text you're sent. Something like:

from logging import LogRecord
from logging.handlers import SysLogHandler

class MultilineSysLogHandler(SysLogHandler):
    def emit(self, record):
        if '\n' in record.msg:
            record_args = [record.args] if isinstance(record.args, dict) else record.args
            for single_line in record.msg.split('\n'):
                single_line_record = LogRecord(
                    name=record.name,
                    level=record.levelno,
                    pathname=record.pathname,
                    msg=single_line,
                    args=record_args,
                    exc_info=record.exc_info,
                    func=record.funcName
                )
                super(MultilineSysLogHandler, self).emit(single_line_record)
        else:
            super(MultilineSysLogHandler, self).emit(record)

Answer 2

OK, figured it out finally...

rsyslog by default escapes all weird characters (ASCII < 32), and this include newlines (as well as tabs and others). Simply add this to your rsyslog config to turn this off:

$EscapeControlCharactersOnReceive off

Answer 3

Alternatively, if you want to keep your syslog intact on one line for parsing, you can just replace the characters when viewing the log.

tail -f /var/log/syslog | sed 's/#012/\n\t/g'