How to log everything that occurs in a Python inte

Question

I'd like to have real-time access to both interpreter input and error and standard output. Preferably this information would be written to a file, so that I can poll the file for changes after every interpreter command has been entered. For example, given an interpreter session:

>>> 5 * 7
35
>>> print("Hello, world!")
Hello, world!
>>> "Hello, world!"
'Hello, world!'

I'd like to see the following in a log file:

> 5 * 7
35
> print("Hello, world!")
Hello, world!
> "Hello, world!"
'Hello, world!'

The formatting is not important; what is important is that I can search the file for key words to trigger interactive events during the session.

What I have learned so far trying to accomplish this:

Python's code module allows me to create an InteractiveConsole object, the raw_input method of which I can redefine to log to a file, like so:

import code
class LoggedConsole(code.InteractiveConsole):
  def __init__(self, locals):
    super(LoggedConsole, self).__init__(locals)
    self.file = open('consolelog.dat', 'a')

  def __del__(self):
    self.file.close()

  def raw_input(self, prompt=""):
    data = input(prompt)
    self.file.write(data+'\n')
    return data

Furthermore, InteractiveConsole uses a built-in write method to log errors, which I can redefine to:

def write(self, data):
  sys.stderr.write(data)
  self.file.write(data+'\n')

I've also learned that the following snippet will log all stdout:

class Tee(object):
  def __init__(self):
    self.file = open('consolelog.dat', 'a')
    self.stdout = sys.stdout

  def __del__(self):
    sys.stdout = self.stdout
    self.file.close()

  def write(self, data):
    self.file.write(data)
    self.stdout.write(data)

sys.stdout = Tee()

My (broken) attempt to bring this all together was to then create a LoggedConsole object, and pass it Tee in locals.

console = LoggedConsole(locals={sys.stdout:LoggedExec()})
console.interact()

(I've not passed locals before, so perhaps I'm doing it incorrectly here, but I don't receive an error.)

Anyways, this will open a new Interactive Console, and will log (after closing) all input and errors, but not output. I've been banging my head against this for a while, and I feel like I'm close, but maybe not even.

Also, is there a way for all of this to occur during the session? Currently all logging takes place once the session is closed.

Thanks for your time, sorry for the wall of text.

edit: I'd like to be able to accomplish this in the standard Python interpreter for portability purposes.

edit2: Jaime's snippet works very well for logging everything I need. Any way, though, that I can have it do so in real time, instead of waiting for the session to close?

edit3: Figured it out :). The final, working snippet:

import code
import sys

class Tee(object):
  def __init__(self, log_fname, mode='a'):
    self.log = open(log_fname, mode)

  def __del__(self):
    # Restore sin, so, se
    sys.stdout = sys.__stdout__
    sys.stdir = sys.__stdin__
    sys.stderr = sys.__stderr__
    self.log.close()

  def write(self, data):
    self.log.write(data)
    self.log.flush()
    sys.__stdout__.write(data)
    sys.__stdout__.flush()

  def readline(self):
    s = sys.__stdin__.readline()
    sys.__stdin__.flush()
    self.log.write(s)
    self.log.flush()
    return s

  def flush(foo):
    return

sys.stdout = sys.stderr = sys.stdin = Tee('consolelog.dat', 'w')

console = code.InteractiveConsole()
console.interact()

Answer 1

You can try and use my logging tool. It's not perfect yet but it solved my problem, which seems similar to your.

https://github.com/hholst80/loginteractive

It works by using LD_PRELOAD to pipe stdin.txt (or $STDIN) to stdout. It works for python and octave, although I have not tested it that much yet.

Answer 2

I've only tested this in python2.7. I don't have 3 handy.

import code
import sys

class Tee(object):

  def __init__(self, log_fname, mode='a'):
    self.log = open(log_fname, mode)

  def __del__(self):
    # Restore sin, so, se
    sys.stdout = sys.__stdout__
    sys.stdir = sys.__stdin__
    sys.stderr = sys.__stderr__
    self.log.close()

  def write(self, data):
    self.log.write(data)
    sys.__stdout__.write(data)

  def readline(self):
    s = sys.__stdin__.readline()
    self.log.write(s)
    return s

# Tie the ins and outs to Tee.
sys.stdout = sys.stderr = sys.stdin = Tee('consolelog.dat', 'w')

console = code.InteractiveConsole()
console.interact()

Answer 3

take a look at IPython (haven't used it myself). Here's a section in the docs that might be of particular interest: http://ipython.org/ipython-doc/dev/interactive/reference.html#session-logging-and-restoring

Answer 4

See this Virtualenv article by Doug Hellmann, showing how to log an iPython session:

If you are comfortable working at the interactive prompt in this way, but want to record what you do for future reference after you close your session, you can use IPython’s logging feature to write the session to a file. To activate the log, use the control command %logstart, as illustrated in Listing 5. The output file is a Python source file, so it is easy to clean it up and turn it into a “real” module when you are done experimenting.

In [6]: %logstart
Activating auto-logging. Current session state plus future input saved.
Filename       : ipython_log.py
Mode           : rotate
Output logging : False
Raw input log  : False
Timestamping   : False
State          : active

In [7]: a = 5

In [8]: b = 6

In [9]: c = a * b

In [10]: c

Out[10]: 30

In [11]: d = [ a, b, c]

In [12]: d

Out[12]: [5, 6, 30]

In [13]: %logstop

Answer 5

You can simply use the unix script command, try:

script -a filename.txt
python
>> print("hi")
hi
>> exit()
exit

filename.txt will record everything you did on that session, it will look something like this:

Script started on Sat Dec 14 11:18:41 2013
python
>> print('hi')
hi
>> exit()
exit

Script done on Sat Dec 14 11:18:59 2013