What is the quickest way to HTTP GET in Python?

Question

What is the quickest way to HTTP GET in Python if I know the content will be a string? I am searching the documentation for a quick one-liner like:

contents = url.get("http://example.com/foo/bar")

But all I can find using Google are httplib and urllib - and I am unable to find a shortcut in those libraries.

Does standard Python 2.5 have a shortcut in some form as above, or should I write a function url_get?

1. I would prefer not to capture the output of shelling out to wget or curl.

Answer 1

theller's solution for wget is really useful, however, i found it does not print out the progress throughout the downloading process. It's perfect if you add one line after the print statement in reporthook.

import sys, urllib

def reporthook(a, b, c):
    print "% 3.1f%% of %d bytes\r" % (min(100, float(a * b) / c * 100), c),
    sys.stdout.flush()
for url in sys.argv[1:]:
    i = url.rfind("/")
    file = url[i+1:]
    print url, "->", file
    urllib.urlretrieve(url, file, reporthook)
print

Answer 2

If you want solution with httplib2 to be oneliner consider instantiating anonymous Http object

import httplib2
resp, content = httplib2.Http().request("http://example.com/foo/bar")

Answer 3

Excellent solutions Xuan, Theller.

For it to work with python 3 make the following changes

import sys, urllib.request

def reporthook(a, b, c):
    print ("% 3.1f%% of %d bytes\r" % (min(100, float(a * b) / c * 100), c))
    sys.stdout.flush()
for url in sys.argv[1:]:
    i = url.rfind("/")
    file = url[i+1:]
    print (url, "->", file)
    urllib.request.urlretrieve(url, file, reporthook)
print

Also, the URL you enter should be preceded by a "http://", otherwise it returns a unknown url type error.

Answer 4

Python 2.x:

import urllib2
contents = urllib2.urlopen("http://example.com/foo/bar").read()

Python 3.x:

import urllib.request
contents = urllib.request.urlopen("http://example.com/foo/bar").read()

Documentation for urllib.request and read.

How is that?

Answer 5

Here is a wget script in Python:

# From python cookbook, 2nd edition, page 487
import sys, urllib

def reporthook(a, b, c):
    print "% 3.1f%% of %d bytes\r" % (min(100, float(a * b) / c * 100), c),
for url in sys.argv[1:]:
    i = url.rfind("/")
    file = url[i+1:]
    print url, "->", file
    urllib.urlretrieve(url, file, reporthook)
print

Answer 6

If you are working with HTTP APIs specifically, there are also more convenient choices such as Nap.

For example, here's how to get gists from Github since May 1st 2014:

from nap.url import Url
api = Url('https://api.github.com')

gists = api.join('gists')
response = gists.get(params={'since': '2014-05-01T00:00:00Z'})
print(response.json())

More examples: https://github.com/kimmobrunfeldt/nap#examples