Python subprocesses don't output properly?

Question

I don't think I'm understanding python subprocess properly at all but here's a simple example to illustrate a point I'm confused about:

#!/usr/bin/env python
import subprocess
lookup_server = subprocess.Popen("nc -l 5050", shell=True)
lookup_client = subprocess.Popen("nc localhost 5050", shell=True, stdin=subprocess.PIPE) 
print lookup_client.poll()
lookup_client.stdin.write("magic\n")
print lookup_client.poll()                                                                        
lookup_client.send_signal(subprocess.signal.SIGINT)
print lookup_client.poll()
lookup_server.wait()
print "Lookup server terminated properly"

The output comes back as

None
None
None

and never completes. Why is this? Also, if I change the first argument of Popen to an array of all of those arguments, none of the nc calls execute properly and the script runs through without ever waiting. Why does that happen?

Ultimately, I'm running into a problem in a much larger program that does something similar using netcat and another program running locally instead of two versions of nc. Either way, I haven't been able to write to or read from them properly. However, when I run them in the python console everything runs as I would expect. All this has me very frustrated. Let me know if you have any insights!

EDIT: I'm running this on Ubuntu Linux 12.04, when I man nc, I get the BSD General Commands manual so I'm assuming this is BSD netcat.

Answer 1

The problem here is that you're sending SIGINT to the process. If you just close the stdin, nc will close its socket and quit, which is what you want.

It sounds like you're actually using nc for the client (although not the server) in your real program, which means you have two easy fixes:

Instead of lookup_client.send_signal(subprocess.signal.SIGINT), just do lookup_client.stdin.close(). nc will see this as an EOF on its input, and exit normally, at which point your server will also exit.

#!/usr/bin/env python
import subprocess
lookup_server = subprocess.Popen("nc -l 5050", shell=True)
lookup_client = subprocess.Popen("nc localhost 5050", shell=True, stdin=subprocess.PIPE) 
print lookup_client.poll()
lookup_client.stdin.write("magic\n")
lookup_client.stdin.close()
print lookup_client.poll()
lookup_server.wait()
print "Lookup server terminated properly"

When I run this, the most common output is:

None
None
magic
Lookup server terminated properly

Occasionally the second None is a 0 instead, and/or it comes after magic instead of before, but otherwise, it's always all four lines. (I'm running on OS X.)

For this simple case (although maybe not your real case), just use communicate instead of trying to do it manually.

#!/usr/bin/env python
import subprocess
lookup_server = subprocess.Popen("nc -l 5050", shell=True)
lookup_client = subprocess.Popen("nc localhost 5050", shell=True, stdin=subprocess.PIPE) 
print lookup_client.communicate("magic\n")
lookup_server.wait()
print "Lookup server terminated properly"

Meanwhile:

Also, if I change the first argument of Popen to an array of all of those arguments, none of the nc calls execute properly and the script runs through without ever waiting. Why does that happen?

As the docs say:

On Unix with shell=True… If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself.

So, subprocess.Popen(["nc", "-l", "5050"], shell=True) does /bin/sh -c 'nc' -l 5050, and sh doesn't know what to do with those arguments.

You probably do want to use an array of args, but then you have to get rid of shell=True—which is a good idea anyway, because the shell isn't helping you here.

One more thing:

lookup_client.send_signal(subprocess.signal.SIGINT)
print lookup_client.poll()

This may print either -2 or None, depending on whether the client has finished responding to the SIGINT and been killed before you poll it. If you want to actually get that -2, you have to call wait rather than poll (or do something else, like loop until poll returns non-None).

Finally, why didn't your original code work? Well, sending SIGINT is asynchronous; there's no guarantee as to when it might take effect. For one example of what could go wrong, it could take effect before the client even opens the socket, in which case the server is still sitting around waiting for a client that never shows up.

You can throw in a time.sleep(5) before the signal call to test this—but obviously that's not a real fix, or even an acceptable hack; it's only useful for testing the problem. What you need to do is not kill the client until it's done everything you want it to do. For complex cases, you'll need to build some mechanism to do that (e.g., reading its stdout), while for simple cases, communicate is already everything you need (and there's no reason to kill the child in the first place).

Answer 2

Your invocation of nc is bad, what will happen if I invoke this as you in command line:

# Server window:
[vyktor@grepfruit ~]$ nc -l 5050

# Client Window
[vyktor@grepfruit ~]$ nc localhost 5050
[vyktor@grepfruit ~]$ echo $?
1

Which mean (1 in $?) failure.

Once you use -p:

-p, --local-port=NUM       local port number

NC starts listening, so:

# Server window
[vyktor@grepfruit ~]$ nc -l -p 5050
    # Keeps handing

# Client window
[vyktor@grepfruit ~]$ echo Hi | nc localhost 5050
    # Keeps hanging

Once you add -c to client invocation:

-c, --close                close connection on EOF from stdin

You'll end up with this:

# Client window
[vyktor@grepfruit ~]$ echo Hi | nc localhost 5050 -c
[vyktor@grepfruit ~]$

# Server window
[vyktor@grepfruit ~]$ nc -l -p 5050
Hi
[vyktor@grepfruit ~]$ 

So you need this python piece of code:

#!/usr/bin/env python
import subprocess
lookup_server = subprocess.Popen("nc -l -p 5050", shell=True)
lookup_client = subprocess.Popen("nc -c localhost 5050", shell=True, 
                      stdin=subprocess.PIPE) 
lookup_client.stdin.write("magic\n")
lookup_client.stdin.close()            # This
lookup_client.send_signal(subprocess.signal.SIGINT) # or this kill
lookup_server.wait()
print "Lookup server terminated properly"