Python: urllib2.HTTPError: HTTP Error 300: Multipl

Question

I have a script that is looking for informations in web text pages and then store them in a dictionary. The script is looking for URL in a list and then process them all in a loop, however it get interrupted in the middle of the process by this error:

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 300: Multiple Choices

I have problems to interpret that and I don't know if there is a way to avoid this kind of problems. Is there a way to add an exception in the script?

Here is my script:

import urllib2
import sys
import re

IDlist = ['C9JVZ1', 'C9JLN0', 'C9J872']  #(there is more than 1500 of them)

URLlist = ["http://www.uniprot.org/uniprot/"+x+".txt" for x in IDlist]

function_list = {}
for id, item in zip(IDlist, URLlist):
    function_list[id] = []
    textfile = urllib2.urlopen(item);
    myfile = textfile.readlines();
    for line in myfile:
        print "line:", line;
        found = re.search('\s[C]:(.+?);', line);
        if found:
            function = found.group(1);
            function_list[id].append(function)

Answer 1

The web server is returning the HTTP status code 300 Multiple Choices (see Wikipedia) for one of the URLs you want to access. This probably means one of the URLs in your list is wrong and the web server wants to help you by providing a list of similar existing URLs.

In general urllib2 turns anything thats not a success or a simple redirection response into an exception and that's what you see there.

When you don't handle an exception somewhere e.g. with a try-except block it usually terminates your program. So you need to wrap your call to urlopen in a try block:

try:
  textfile = urllib2.urlopen(item);
except urllib2.HTTPError:
  # Do something here to handle the error. For example:
  print("URL", item, "could not be read.")
  continue