I am using LWP to get the content from some sites and whenever the page size is larger than one megabyte, I would like LWP to stop downloading the page. This is accomplished with this code:
my $ua = LWP::UserAgent->new;
$ua->max_size(1024);
And it works fine. The only problem is that even when the page is over 1mb, I would still like to be able to get the content of what it has downloaded so far. However, whenever I do this (the traditional way to get content with LWP when there are no errors):
$ua->content;
It seems to not be the actual page's content but a bunch of nonsense. Does anyone know of a way to set max_size and still get whatever was downloaded before it hit the limit with LWP or even WWW::Mechanize? Thanks a lot!
