handle lwp timeout effectively

Question

I am using LWP to download content from web pages, and I would like to limit the amount of time it waits for a page. This is accomplished in lwp like this:

my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->get($url);

And this works fine, except for whenever the timeout reaches its limit, it just dies and I can't continue on with the script! I'd really like to handle this timeout properly so that I can record that the url had a timeout and then move on to my next one. Does anyone know how to do this? Thanks!

Answer 1

You can do the equivalent of a try{} catch {} in Perl using eval blocks:

http://perldoc.perl.org/functions/eval.html

Answer 2

For most purposes, LWP::UserAgent's timeout is sufficient, but it does suffer some drawbacks… it applies to each system call, rather than to the aggregate of them. If you truly need a fixed timeout period, this is one of the things that LWPx::ParanoidAgent takes care off.

Answer 3

LWP::Agent's get() returns a HTTP::Response object that you can use for checking errors:

use LWP::Agent;
use HTTP::Status ();

my $ua = LWP::UserAgent->new;
$ua->timeout(10);
my $response = $ua->get($url);

if ($response->is_error) {
    printf "[%d] %s\n", $response->code, $response->message;

    # record the timeout
    if ($response->code == HTTP::Status::HTTP_REQUEST_TIMEOUT) {
        ...
    }
}

Btw, the better practice nowadays is to use Try::Tiny instead of eval {...}. It gives you try {...} catch {...}. and it resolves some problems with checking if $@ (see the background section in the Try::Tiny documentation).