Issues with LWP when using HTTP/1.1: bad chunk-siz

Question

I've been writing some Perl utilities to interact with some of my company's internal web sites. A while back I started encountering an intermittent error when accessing certain pages: "Bad chunk-size in HTTP response". I tracked this message down to the module Net::HTTP::Methods. I spent a little time trying to diagnose the ultimate origin of the error, but without success. Then somewhere I found a suggestion that setting the protocol of my HTTP requests to "HTTP/1.0" might help, and to my surprise, it did. Once I did this for all of my page requests that had been causing me problems, it was smooth sailing.

More recently, I've been writing code that interacts with our local Bugzilla server. There's an intermittent error where the HTTP responses are truncated somewhere in the middle of the page. I was perplexed at first, but then I remembered the problem I described above and tried setting the protocol to HTTP/1.0. Again, it worked. I tried accessing Bugzilla's "Enter CR" page a hundred times. Using the default HTTP/1.1, I got a truncation of the response six times; with HTTP/1.0, zero times.

Does anyone know what could be happening here? I never see page truncation in my browser, which suggests that the problem is with the LWP library, but it's hard to believe that such an ancient and widely-used library could have such a glaring bug. I run Ubuntu with up-to-date Perl modules, and the Bugzilla responses identify the server as Apache, if that helps at all.

Answer 1

It is really hard from your description to see what is going on, but

• if you enforce HTTP/1.0 the server is not allowed to send data with Transfer-Encoding: chunked, so this might explain why it works with HTTP/1.0
• Ubuntu does not provide the latest version of LWP, they provide mostly only updates to the version they've shipped originally. And which version that is depends on your Ubuntu version. You might check the installed version with perl -MLWP::UserAgent -e 'warn LWP::UserAgent->VERSION'. Current version is 6.06.
• It is impossible to debug the problem without getting more information. Essential information are the version of LWP. A packet capture (tcpdump with real content, i.e. tcpdump -s0 -wfile ...) of the failed connection would be very helpful too to see the data the server is sending.