What is the quickest way to HTTP GET in Python?

Question

What is the quickest way to HTTP GET in Python if I know the content will be a string? I am searching the documentation for a quick one-liner like:

contents = url.get("http://example.com/foo/bar")

But all I can find using Google are httplib and urllib - and I am unable to find a shortcut in those libraries.

Does standard Python 2.5 have a shortcut in some form as above, or should I write a function url_get?

1. I would prefer not to capture the output of shelling out to wget or curl.

Answer 1

Have a look at httplib2, which - next to a lot of very useful features - provides exactly what you want.

import httplib2

resp, content = httplib2.Http().request("http://example.com/foo/bar")

Where content would be the response body (as a string), and resp would contain the status and response headers.

It doesn't come included with a standard python install though (but it only requires standard python), but it's definitely worth checking out.

Answer 2

You could use a library called requests.

import requests
r = requests.get("http://example.com/foo/bar")

This is quite easy. Then you can do like this:

>>> print(r.status_code)
>>> print(r.headers)
>>> print(r.content)

Answer 3

Without further necessary imports this solution works (for me) - also with https:

try:
    import urllib2 as urlreq # Python 2.x
except:
    import urllib.request as urlreq # Python 3.x
req = urlreq.Request("http://example.com/foo/bar")
req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36')
urlreq.urlopen(req).read()

I often have difficulty grabbing the content when not specifying a "User-Agent" in the header information. Then usually the requests are cancelled with something like: urllib2.HTTPError: HTTP Error 403: Forbidden or urllib.error.HTTPError: HTTP Error 403: Forbidden.

Answer 4

How to also send headers

Python 3:

import urllib.request
contents = urllib.request.urlopen(urllib.request.Request(
    "https://api.github.com/repos/cirosantilli/linux-kernel-module-cheat/releases/latest",
    headers={"Accept" : 'application/vnd.github.full+json"text/html'}
)).read()
print(contents)

Python 2:

import urllib2
contents = urllib2.urlopen(urllib2.Request(
    "https://api.github.com",
    headers={"Accept" : 'application/vnd.github.full+json"text/html'}
)).read()
print(contents)