Short version (if you can answer the short version it does the job for me, the rest is mainly for the benefit of other people with a similar task):
In python in Windows, I want to create 2 file objects, attached to the same file (it doesn't have to be an actual file on the hard-drive), one for reading and one for writing, such that if the reading end tries to read it will never get EOF (it will just block until something is written). I think in linux os.mkfifo() would do the job, but in Windows it doesn't exist. What can be done? (I must use file-objects).
Some extra details:
I have a python module (not written by me) that plays a certain game through stdin and stdout (using raw_input() and print). I also have a Windows executable playing the same game, through stdin and stdout as well. I want to make them play one against the other, and log all their communication.
Here's the code I can write (the get_fifo() function is not implemented, because that's what I don't know to do it Windows):
class Pusher(Thread):
def __init__(self, source, dest, p1, name):
Thread.__init__(self)
self.source = source
self.dest = dest
self.name = name
self.p1 = p1
def run(self):
while (self.p1.poll()==None) and\
(not self.source.closed) and (not self.source.closed):
line = self.source.readline()
logging.info('%s: %s' % (self.name, line[:-1]))
self.dest.write(line)
self.dest.flush()
exe_to_pythonmodule_reader, exe_to_pythonmodule_writer =\
get_fifo()
pythonmodule_to_exe_reader, pythonmodule_to_exe_writer =\
get_fifo()
p1 = subprocess.Popen(exe, shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
old_stdin = sys.stdin
old_stdout = sys.stdout
sys.stdin = exe_to_pythonmodule_reader
sys.stdout = pythonmodule_to_exe_writer
push1 = Pusher(p1.stdout, exe_to_pythonmodule_writer, p1, '1')
push2 = Pusher(pythonmodule_to_exe_reader, p1.stdin, p1, '2')
push1.start()
push2.start()
ret = pythonmodule.play()
sys.stdin = old_stdin
sys.stdout = old_stdout
Following the two answers above, I accidentally bumped into the answer. os.pipe() does the job. Thank you for your answers.
I'm posting the complete code in case someone else is looking for this:
import subprocess
from threading import Thread
import time
import sys
import logging
import tempfile
import os
import game_playing_module
class Pusher(Thread):
def __init__(self, source, dest, proc, name):
Thread.__init__(self)
self.source = source
self.dest = dest
self.name = name
self.proc = proc
def run(self):
while (self.proc.poll()==None) and\
(not self.source.closed) and (not self.dest.closed):
line = self.source.readline()
logging.info('%s: %s' % (self.name, line[:-1]))
self.dest.write(line)
self.dest.flush()
def get_reader_writer():
fd_read, fd_write = os.pipe()
return os.fdopen(fd_read, 'r'), os.fdopen(fd_write, 'w')
def connect(exe):
logging.basicConfig(level=logging.DEBUG,\
format='%(message)s',\
filename=LOG_FILE_NAME,
filemode='w')
program_to_grader_reader, program_to_grader_writer =\
get_reader_writer()
grader_to_program_reader, grader_to_program_writer =\
get_reader_writer()
p1 = subprocess.Popen(exe, shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
old_stdin = sys.stdin
old_stdout = sys.stdout
sys.stdin = program_to_grader_reader
sys.stdout = grader_to_program_writer
push1 = Pusher(p1.stdout, program_to_grader_writer, p1, '1')
push2 = Pusher(grader_to_program_reader, p1.stdin, p1, '2')
push1.start()
push2.start()
game_playing_module.play()
sys.stdin = old_stdin
sys.stdout = old_stdout
fil = file(LOG_FILE, 'r')
data = fil.read()
fil.close()
return data
if __name__=='__main__':
if len(sys.argv) != 2:
print 'Usage: connect.py exe'
print sys.argv
exit()
print sys.argv
print connect(sys.argv[1])
On Windows, you are looking at (Named or Anonymous) Pipes.
A pipe is a section of shared memory that processes use for communication. The process that creates a pipe is the pipe server. A process that connects to a pipe is a pipe client. One process writes information to the pipe, then the other process reads the information from the pipe.
To work with Windows Pipes, you can use Python for Windows extensions (pywin32), or the Ctypes module. A special utility module, win32pipe, provides an interface to the win32 pipe API's. It includes implementations of the popen[234]() convenience functions.
See how-to-use-win32-apis-with-python and similar SO questions (not specific to Pipes, but points to useful info).
For a cross-platform solution, I'd recommend building the file-like object on top of a socket on localhost (127.0.0.1) -- that's what IDLE does by default to solve a problem that's quite similar to yours.
os.pipe() returns an anonymous pipe, or a named pipe on Windows, which is very lightweight and efficient.
TCP sockets (as suggested by user1495323) are more heavyweight: you can see them with netstat for example, and each one requires a port number, and the number of available ports is limited to 64k per peer (e.g. 64k from localhost to localhost).
On the other hand, named pipes (on Windows) are limited because:
- You can't use select() for nonblocking I/O on Windows, because they're not sockets.
- There's no apparent way to
read() with a timeout, and
- Even making them non-blocking is difficult.
And sockets can be wrapped in Python-compatible filehandles using makefile(), which allows them to be used to redirect stdout or stderr. This makes this an attractive option for some use cases, such as sending stdout from one thread to another.
A socket can be constructed with an automatically-assigned port number like this (based on the excellent Python socket HOWTO):
with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as input_socket:
# Avoid socket exhaustion by setting SO_REUSEADDR <https://stackoverflow.com/a/12362623/648162>:
input_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
# localhost doesn't work if the definition is missing from the hosts file,
# and 127.0.0.1 only works with IPv4 loopback, but socket.gethostname()
# should always work:
input_socket.bind((socket.gethostname(), 0))
random_port_number = input_socket.getsockname()[1]
input_socket.listen(1)
# Do something with input_socket, for example pass it to another thread.
output_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# close() should not strictly be necessary here, but since connect() could fail, it avoids leaking fds
# in that case. "If a file descriptor is given, it is closed when the returned I/O object is closed".
with output_socket:
output_socket.connect((socket.gethostname(), random_port_number))
The user of input_socket (e.g. another thread) can then do:
with input_socket:
while True:
readables, _, _ = select.select([input_socket], [], [input_socket], 1.0)
if len(readables) > 0:
input_conn, addr = self.input_socket.accept()
break
with input_conn:
while True:
readables, _, errored = select.select([input_conn], [], [input_conn], 1.0)
if len(errored) > 0:
print("connection errored, stopping")
break
if len(readables) > 0:
read_data = input_conn.recv(1024)
if len(read_data) == 0:
print("connection closed, stopping")
break
else:
print(f"read data: {read_data!r}")
