Watch long term process in Ipython notebook

问题:

For now, after reconnect ipython notebook is not able to print any output from running kernel. I tried following work-around:

import sys, time

with open('foo.log','w') as sys.stdout:
  for i in range(5):
    print i
    time.sleep(1)

everything is nice, I can see the process with tail -f foo.log, but at the end whole thing crashes and I get following error:

ERROR:tornado.general:Uncaught exception, closing connection.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 407, in _run_callback
    callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 260, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 218, in dispatch_shell
    sys.stdout.flush()
ValueError: I/O operation on closed file
ERROR:tornado.general:Uncaught exception, closing connection.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 433, in _handle_events
    self._handle_recv()
  File "/usr/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 465, in _handle_recv
    self._run_callback(callback, msg)
  File "/usr/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 407, in _run_callback
    callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 260, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 218, in dispatch_shell
    sys.stdout.flush()
ValueError: I/O operation on closed file
ERROR:tornado.application:Exception in callback None
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 883, in start
    handler_func(fd_obj, events)
  File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 433, in _handle_events
    self._handle_recv()
  File "/usr/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 465, in _handle_recv
    self._run_callback(callback, msg)
  File "/usr/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 407, in _run_callback
    callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 260, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 218, in dispatch_shell
    sys.stdout.flush()
ValueError: I/O operation on closed file

回答1:

I have finally managed to log messages from jupyter into file and flush memory immediately:

class InstantLogger():
  def __init__(self, f):
    self.f = open(f,'w')
  def __getattr__(self,name):
    return object.__getattribute__(self.f, name)
  def write(self, x):
    self.f.write(x)
    self.f.flush()
  def flush(self):
    self.f.flush()
  def close(self):
    self.f.close()

than I can use it as:

old_stdout = sys.stdout
sys.stdout = InstantLogger('file.log')
print ....
sys.stdout.close()
sys.stdout = old_stdout