What is best module for parallel processing in Per

Question

What is best module for parallel process in Perl? I have never done the parallel processing in Perl.

What is good Perl module for parallel process which is going to used in DB access and mailing ?

I have looked in module Parallel::ForkManager. Any idea appreciated.

Answer 1

Many-core Engine for Perl (MCE) has been posted on CPAN.

http://code.google.com/p/many-core-engine-perl/

https://metacpan.org/module/MCE

MCE comes with various examples showing real-world use case scenarios on parallelizing something as small as cat (try with -n) to greping for patterns and word count aggregation.

barrier_sync.pl
          A barrier sync demonstration.

cat.pl    Concatenation script, similar to the cat binary.
egrep.pl  Egrep script, similar to the egrep binary.
wc.pl     Word count script, similar to the wc binary.

findnull.pl
          A parallel driven script to report lines containing
          null fields. It is many times faster than the binary
          egrep command. Try against a large file containing
          very long lines.

flow_model.pl
          Demonstrates MCE::Flow, MCE::Queue, and MCE->gather.

foreach.pl, forseq.pl, forchunk.pl
          These take the same sqrt example from Parallel::Loops
          and measures the overhead of the engine. The number
          indicates the size of @input which can be submitted
          and results displayed in 1 second.

          Parallel::Loops:     600  Forking each @input is expensive
          MCE foreach....:  34,000  Sends result after each @input
          MCE forseq.....:  70,000  Loops through sequence of numbers
          MCE forchunk...: 480,000  Chunking reduces overhead

interval.pl
          Demonstration of the interval option appearing in MCE 1.5.

matmult/matmult_base.pl, matmult_mce.pl, strassen_mce.pl
          Various matrix multiplication demonstrations benchmarking
          PDL, PDL + MCE, as well as parallelizing Strassen
          divide-and-conquer algorithm. Also included are 2 plain
          Perl examples.

scaling_pings.pl
          Perform ping test and report back failing IPs to
          standard output.

seq_demo.pl
          A demonstration of the new sequence option appearing
          in MCE 1.3. Run with seq_demo.pl | sort

tbray/wf_mce1.pl, wf_mce2.pl, wf_mce3.pl
          An implementation of wide finder utilizing MCE.
          As fast as MMAP IO when file resides in OS FS cache.
          2x ~ 3x faster when reading directly from disk.

Answer 2

Parallel::ForkManager, as the POD says, can limit the number of processes forked off. You could then use the children to do any work. I remember using the module for a downloader.

Answer 3

You could also look at threads, Coro, Parallel::Iterator.

Answer 4

Well, it all depends on particular case. Depending on what you exactly need to achieve, you might be better of with one of following:

• Parallel::TaskManager
• POE
• AnyEvent

For some tasks there are also specialized modules - for example LWP::Parallel::UserAgent, which basically means - you have to give us much more details about what you want to achieve to get best possible answer.