Python 2.6 send connection object over Queue / Pip

Question

Given this bug (Python Issue 4892) that gives rise to the following error:

>>> import multiprocessing
>>> multiprocessing.allow_connection_pickling()
>>> q = multiprocessing.Queue()
>>> p = multiprocessing.Pipe()
>>> q.put(p)
>>> q.get()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File     "/.../python2.6/multiprocessing/queues.py", line 91, in get
    res = self._recv()
TypeError: Required argument 'handle' (pos 1) not found

Does anyone know of a workaround to pass a Connection object on a Queue?

Thank you.

Answer 1

Here's roughly what I did:

# Producer
from multiprocessing.reduction import reduce_connection
from multiprocessing import Pipe

   # Producer and Consumer share the Queue we call queue
def handle(queue):
   reader, writer = Pipe()
   pickled_writer = pickle.dumps(reduce_connection(writer))
   queue.put(pickled_writer)

and

# Consumer
from multiprocessing.reduction import rebuild_connection

def wait_for_request():
    pickled_write = queue.get(block=True) # block=True isn't necessary, of course
    upw = pickle.loads(pickled_writer) # unpickled writer
    writer = upw[0](upw[1][0],upw[1][1],upw[1][2])

The last line is cryptic, coming from the following:

>>> upw
(<function rebuild_connection at 0x1005df140>,
(('/var/folders/.../pymp-VhT3wX/listener-FKMB0W',
17, False), True, True))

Hope that helps someone else. It works fine for me.

Answer 2

(What I believe is) A better method, after some playing around (I was having the same problem. Wanted to pass a pipe through a pipe.) before discovering this post:

>>> from multiprocessing import Pipe, reduction
>>> i, o = Pipe()
>>> reduced = reduction.reduce_connection(i)
>>> newi = reduced[0](*reduced[1])
>>> newi.send("hi")
>>> o.recv()
'hi'

I'm not entirely sure why this is built this way (someone would need insight into what the heck the reduction part of multiprocessing is about for that) but it does definitely work, and requires no pickle import. Other than that, it's pretty close to the above in what it does, but simpler. I also threw this into the python bug report so others know of the workaround.