Python: How to combine a process poll and a non-bl

Question

I have an idea. Write a WebSocket based RPC that would process messages according to the scenario below.

1. Client connects to a WS (web socket) server
2. Client sends a message to the WS server
3. WS server puts the message into the incoming queue (can be a multiprocessing.Queue or RabbitMQ queue)
4. One of the workers in the process pool picks up the message for processing
5. Message is being processed (can be blazingly fast or extremely slow - it is irrelevant for the WS server)
6. After the message is processed, results of the processing are pushed to the outcoming queue
7. WS server pops the result from the queue and sends it to the client

NOTE: the key point is that the WS server should be non-blocking and responsible only for:

• connection acceptance
• getting messages from the client and puting them into the incoming queue
• popping messages from the outcoming queue and sending them back to the client

NOTE2: it might be a good idea to store client identifier somehow and pass it around with the message from the client

NOTE3: it is completely fine that because of queueing the messages back and forth the speed of simple message processing (e.g. get message as input and push it back as a result) shall become lower. Target goal is to be able to run processor expensive operations (rough non-practical example: several nested “for” loops) in the pool with the same code style as handling fast messages. I.e. pop message from the input queue together with some sort of client identifier, process it (might take a while) and push the processing results together with client ID to the output queue.

Questions:

• In TornadoWeb, if I have a queue (multiprocessing or Rabit), how can I make Tornado’s IOLoop trigger some callback whenever there is a new item in that queue? Can you navigate me to some existing implementation if there is any?
• Is there any ready implementation of such a design? (Not necessarily with Tornado)
• Maybe I should use another language (not python) to implement such a design?

Acknowledgments:

• Recommendations to use REST and WSGI for whatever goal I aim to achieve are not welcome
• Comments like “Here is a link to the code that I found by googling for 2 seconds. It has some imports from tornado and multiprocessing.I am not sure what it does, however I am for 99% certain that it isexactly what you need” are not welcome neither
• Recommendations to use asynchronous libraries instead of normal blocking ones are ... :)

Answer 1

Tornado's IOLoop allows you handling events from any file object by its file descriptor, so you could try this:

• connect with each of your workers processes through multiprocessing.Pipe
• call add_handler for each pipe's parent end (using the connection's fileno())
• make the workers write some random garbage each time they put something into the output queue, no matter if that's multiprocessing.Queue of any MQ.
• handle the answers form the workers in the event handlers