Currently I am using zeromq with pub-sub pattern, I have single worker to publish and many(8) subscriber (all will subscribe) to same pattern.
Now I tried multiprocessing to spawn subscribers it works. I am missing few messages.
Why I am using multiprocessing is to handle each message as it arrives and process them, every second publisher publishes 10 - 100 messages.
In this case it is advised use multiprocessing or gevent ?
Multiprocessing will clearly have much higher memory overhead but will utilize another core (and you don't have to worry about lack of pre-emption) so.. it depends on your needs.
It's likely that multiple processes using gevent will get you the highest throughput / lowest latency.
I am answering my own question
I choose multiprocessing over gevent
- Server had 8 cores
- Parallelism was needed
- Choose ZMQ over multiprocessing queue or gevent queue.
