倾倒multiprocessing.Queue到一个列表(Dumping a multiproces

我想转储multiprocessing.Queue到一个列表。 对于这项任务，我写了下面的功能：

import Queue

def dump_queue(queue):
    """
    Empties all pending items in a queue and returns them in a list.
    """
    result = []

    # START DEBUG CODE
    initial_size = queue.qsize()
    print("Queue has %s items initially." % initial_size)
    # END DEBUG CODE

    while True:
        try:
            thing = queue.get(block=False)
            result.append(thing)
        except Queue.Empty:

            # START DEBUG CODE
            current_size = queue.qsize()
            total_size = current_size + len(result)
            print("Dumping complete:")
            if current_size == initial_size:
                print("No items were added to the queue.")
            else:
                print("%s items were added to the queue." % \
                      (total_size - initial_size))
            print("Extracted %s items from the queue, queue has %s items \
            left" % (len(result), current_size))
            # END DEBUG CODE

            return result

但是，由于某种原因，这是行不通的。

遵守以下shell会话：

>>> import multiprocessing
>>> q = multiprocessing.Queue()
>>> for i in range(100):
...     q.put([range(200) for j in range(100)])
... 
>>> q.qsize()
100
>>> l=dump_queue(q)
Queue has 100 items initially.
Dumping complete:
0 items were added to the queue.
Extracted 1 items from the queue, queue has 99 items left
>>> l=dump_queue(q)
Queue has 99 items initially.
Dumping complete:
0 items were added to the queue.
Extracted 3 items from the queue, queue has 96 items left
>>> l=dump_queue(q)
Queue has 96 items initially.
Dumping complete:
0 items were added to the queue.
Extracted 1 items from the queue, queue has 95 items left
>>> 

这里发生了什么事？ 为什么不是所有的物品被倾倒？

Answer 1:

试试这个：

import Queue
import time

def dump_queue(queue):
    """
    Empties all pending items in a queue and returns them in a list.
    """
    result = []

    for i in iter(queue.get, 'STOP'):
        result.append(i)
    time.sleep(.1)
    return result

import multiprocessing
q = multiprocessing.Queue()
for i in range(100):
    q.put([range(200) for j in range(100)])
q.put('STOP')
l=dump_queue(q)
print len(l)

多处理队列具有其具有馈线螺纹，其拉断工作的缓冲液，并将其冲洗到管道的内部缓冲器。 如果不是所有的对象都被刷新，我可以看到空过早上升的情况下。 使用定点指示队列的末尾是安全的（可靠）。 此外，使用ITER（GET，定点）成语就是不是靠空好。

我不喜欢，它可能是由于冲洗时间提高空（我加了time.sleep（0.1），使上下文切换到馈线线程，你可能不需要它，它的工作原理没有它 - 这是一个习惯释放GIL）。

Answer 2:

在某些情况下，我们已经计算的一切，我们只是传递队列转换。

shared_queue = Queue()
shared_queue_list = []
...
join() #All process are joined
while shared_queue.qsize() != 0:
    shared_queue_list.append(shared_queue.get())

现在shared_queue_list具有转换到一个列表的结果。

文章来源: Dumping a multiprocessing.Queue into a list