Limiting response size with httplib2

Question

Is it possible to limit the response size with httplib2? For instance if it sees an HTTP body over X bytes the connection will just close without consuming more bandwidth. Or perhaps only download the first X bytes of a file.

Answer 1

Assuming that the server is sending the response body size in the Content-Length response header field, you can do it yourself.

First, call Http.request(method="HEAD") to retrieve only the headers and not the body. Then inspect the Content-Length field of the response to see if it is below your threshold. If it is, make a second request with the proper GET or POST method to retrieve the body; otherwise produce an error.

If the server isn't giving you the Content-Length (or is lying about it), it doesn't look like there is a way to cut off the download after some number of bytes.