How do I make parallel HTTP requests in Perl, and

Question

Using Perl, I'm looking for a simple way to perform a handful of HTTP requests in parallel, where I get responses back in the same order I sent them after they complete, e.g.:

my ($google, $perl) = foobar(GET => 'http://www.google.com/',
                             GET => 'http://www.perl.org/');

Is there a module I should be looking at?

I know I can do the bookkeeping by hand, but I feel spoiled after being able to do this using jQuery's when method, and I'd love to have as simple a solution using Perl.

Thanks for your help.

Answer 1

use threads;
use LWP::UserAgent qw( );

my $ua = LWP::UserAgent->new();
my @threads;
for my $url ('http://www.google.com/', 'http://www.perl.org/') {
   push @threads, async { $ua->get($url) };
}

for my $thread (@threads) {
   my $response = $thread->join;
   ...
}

The best part is that the parent doesn't wait for all requests to be completed. As soon as the right request is completed, the parent will unblock to process it.

---

If you used Parallel::ForkManager or something else where you can't wait for a specific child, you can use the following code to order the results:

for my $id (0..$#urls) {
   create_task($id, $urls[$id]);
}

my %responses;
for my $id (0..$#urls) {
   if (!exists($responses{$id})) {
      my ($id, $response) = wait_for_a_child_to_complete();
      $responses{$id} = $response;
      redo;
   }

   my $response = delete($responses{$id});
   ...
}

Answer 2

I am a fan of Mojo! From the Mojo::UserAgent documentation:

use Mojo;
use Mojo::UserAgent;
# Parallel requests
my $ua = Mojo::UserAgent->new;
$ua->max_redirects(5);
my $delay = Mojo::IOLoop->delay;
for my $url ('http://www.google.com/', 'http://www.perl.org/') {
  $delay->begin;
  $ua->get($url => sub {
    my ($ua, $tx) = @_;
    $delay->end($tx->res->dom);
  });
}
my @responses = $delay->wait;
print join "\n", @responses

Enjoy!

EDIT

Btw. you do not have to process the responses at the end, you may do it in between:

# ...
$ua->get($url => sub {
    my ($ua, $tx) = @_;
    $delay->end(1);
    # process $tx->res here
});
# ...
$delay->wait;