Asynchronously retrieving information from a proce

Question

Here is the pseudo code for what I want to do.

import time

def run():

    while x < 10000000:
        x += 1

if __name__ == "__main__":

    p = Process(run)
    p.start()

    time.sleep(3)

    #some code that I don't know that will give me the current value of x

Pythons threading module seems to be the way to go however I have yet to successfully implement this example.

Answer 1

It really depends on what you're trying to accomplish and the frequency of creation and memory usage of your subprocesses. A few long-lived ones, and you can easily get away with multiple OS-level processes (see the subprocess module`). If you're spawning a lot of little ones, threading is faster and has less memory overhead. But with threading you run into problems like "thread safety", the global interpreter lock, and nasty, boring stuff like semaphores and deadlocks.

Data sharing strategies between two processes or threads can be roughly divided into two categories: "Let's share a block of memory" (using Locks and Mutexes) and "Let's share copies of data" (using messaging, pipes, or sockets). The sharing method is light on memory, but difficult to manage because it means ensuring that one thread doesn't read the same part of shared memory as another thread is writing to it, which is not trivial and hard to debug. The copying method is heavier on memory, but easier to make sense of. Also, it has the distinct advantage of being able to be pretty trivially ported to a network, allowing for distributed computing.

You'll also have to think about the underlying OS. I don't know the specifics, but some are better than others at different approaches.

I'd say start with something like RabbitMQ.

Answer 2

Well here it is

from multiprocessing import Process, Pipe
import time

def f(conn):
    x = 0
    while x < 10000000:
        if conn.poll():
            if conn.recv() == "get":
                conn.send(x)

        x += 1

    conn.close()

if __name__ == '__main__':

    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(child_conn,))
    p.start()

    time.sleep(2)
    parent_conn.send("get")
    print(parent_conn.recv())

    p.join()

turned out to be a duplicate, my version is just more generic.

Answer 3

Everything you need is in the multiprocessing module. Perhaps a shared memory object would help here?

Note that threading in Python is affected by the Global Interpreter Lock, which essentially prevents multithreaded Python code.