IPython won't capture some command outputs (e.

I don't understand why IPython does not assign the result of some system command to python variables. This seems to constantly happen to me to the ack and ag executables

For example, the following command produces output:

In [1]: !ack --nocolor foo
bar
1:foo

However, whenever I save that result to a variable, I get an empty output

In [2]: out=!ack --nocolor foo

In [3]: out
Out[3]: []

I get this problem even when I try all sorts of hacks:

In [4]: out=!ack --nocolor foo > tmp; sleep 1; cat tmp

In [5]: out
Out[5]: []

In fact, tmp is empty in the last case, which suggests that the output capture messes up with these commands.

Does anyone how if this is a problem with IPython or ack/ag, or simply my misunderstanding of how IPython should behave here?

I've deduced that out = !cmd uses %sx. This is different from how !cmd is run (see docs for %sw and %system).

%sx goes through several layers of functions, and ends up calling

# import IPython
IPython.utils._process_common.process_handler

Its code is similar to the subprocess call that @Elliott Frisch uses in his deleted answer:

p = subprocess.Popen("ack --nocolor foo", stdout=subprocess.PIPE, shell=True)
(output, err) = p.communicate()

I abstracted the process_handler code in:

def cmd1(astr='ack --nocolor 15 *.txt'):
    callback = lambda p: p.communicate()
    stderr = subprocess.PIPE
    stderr = subprocess.STDOUT
    shell = True
    close_fds = True
    executable = None
    p = subprocess.Popen(astr, 
                         shell=shell,
                         executable=executable,
                         #stdin=subprocess.PIPE,
                         stdout=subprocess.PIPE,
                         stderr=stderr,
                         close_fds=close_fds,
                         )
    out = callback(p)
    return out, p.returncode

This works:

In [40]: cmd1()
Out[40]: 
((b'stack53269737.txt:2:11 12 13 14 15 16\ntest.txt:3:11\t12 13 14 15\ntest1.txt:5:  0.054181,  0.506962,  0.315159,  0.653104\n',
  None),
 0)

But if I uncomment the stdin line, it fails:

In [42]: cmd1()
Out[42]: ((b'', None), 1)

So it's the

stdin=subprocess.PIPE,

parameter that causes the ack call to fail. It doesn't cause problems with other common shell commands like ls or grep.

ack help has:

 --[no]filter               Force ack to treat standard input as a pipe
                            (--filter) or tty (--nofilter)

Adding --nofilter to my commands (--nocolor isn't needed with this redirection):

In [50]: cmd1('ack --nofilter 15 *.txt')
Out[50]: 
((b'stack53269737.txt:2:11 12 13 14 15 16\ntest.txt:3:11\t12 13 14 15\ntest1.txt:5:  0.054181,  0.506962,  0.315159,  0.653104\n',
  None),
 0)

In [51]: out = !ack --nofilter 15 *.txt
In [52]: out
Out[52]: 
['stack53269737.txt:2:11 12 13 14 15 16',
 'test1.txt:5:  0.054181,  0.506962,  0.315159,  0.653104',
 'test.txt:3:11\t12 13 14 15']

So that's the key - force ack to ignore the piped input (though I don't fully understand the details).