Is there a way to redirect stderr to file in Jupyt

Question

There was a redirect_output function in IPython.utils, and there was a %%capture magic function, but these are now gone, and this thread on the topic is now outdated.

I'd like to do something like the following:

from IPython.utils import io
from __future__ import print_function
with io.redirect_output(stdout=False, stderr="stderr_test.txt"):
    while True:
        print('hello!', file=sys.stderr)

Thoughts? For more context, I am trying to capture the output of some ML functions that run for hours or days, and output a line every 5-10 seconds to stderr. I then want to take the output, munge it, and plot the data.

Answer 1

@Ben, just replacing sys.stderr did not work, and the full flush logic suggested in the post was necessary. But thank you for the pointer as it finally gave me a working version:

import sys
oldstderr = sys.stderr
sys.stderr = open('log.txt', 'w')
class flushfile():
    def __init__(self, f):
        self.f = f
    def __getattr__(self,name): 
        return object.__getattribute__(self.f, name)
    def write(self, x):
        self.f.write(x)
        self.f.flush()
    def flush(self):
        self.f.flush()
sys.sterr = flushfile(sys.stderr)
from __future__ import print_function
# some long running function here, e.g. 
for i in range(1000000):
    print('hello!', file=sys.stderr)
sys.stderr = oldstderr

It would have been nice if Jupyter kept the redirect_output() function and/or the %%capture magic.

Answer 2

You could probably try replacing sys.stderr with some other file descriptor the same way as suggested here.

import sys
oldstderr = sys.stderr
sys.stderr = open('log.txt', 'w')
# do something
sys.stderr = oldstderr