Interprocess communication with SPSC queue in pyth

问题:

I have multiple write-heavy Python applications (producer1.py, producer2.py, ...) and I'd like to implement an asynchronous, non-blocking writer (consumer.py) as a separate process, so that the producers are not blocked by disk access or contention.

To make this more easily optimizable, assume I just need to expose a logging call that passes a fixed length string from a producer to the writer, and the written file does not need to be sorted by call time. And the target platform can be Linux-only. How should I implement this with minimal latency penalty on the calling thread?

This seems like an ideal setup for multiple lock-free SPSC queues but I couldn't find any Python implementations.

---

Edit 1

I could implement a circular buffer as a memory-mapped file on /dev/shm, but I'm not sure if I'll have atomic CAS in Python?

回答1:

The simplest way would be using an async TCP/Unix Socket server in consumer.py.
Using HTTP will be an overhead in this case.

A producer, TCP/Unix Socket client, will send data to consumer then consumer will respond right away before writing data in disk drive.

File IO in consumer are blocking but it will not block producers as stated above.