A queueing system for Perl

问题:

I'm working on a Perl project which needs a FIFO message queue for distributing tasks between several processes on a single machine (UNIX). The queue size may grow up to 1M jobs.

I've tried IPC::DirQueue, but it becomes awfully slow with 50k or so jobs enqueued. What are good alternatives to this module which can be used in Perl?

回答1:

I've had pretty good success with using ZeroMQ for this sort of problem, both with Perl and other languages.

In my experience, the ZeroMQ module appears to be the most reliable binding for Perl currently.

回答2:

I haven't tried it under the conditions you list, but Thread::Queue has proven useful to me. Combined with forks, it can be used to communicate with processes, as long as those processes were spawned by the queue creator.

use forks;  # If you want to use processes instead of threads.
use Thread::Queue qw( );

A worker model is usually ideal.

my $q = Thread::Queue->new();

my @workers;
for (1..$NUM_WORKERS) {
   push @workers, async {
      while (my $item = $q->dequeue()) {
         ...
      }
   };
}

# ... Enqueue requests [ $q->enqueue($request); ] ...

# Signal termination
$q->enqueue(undef) for 1..@workers;

# Collect workers.
$_->join() for @workers;