HTTP basic authentication not working in python 3.

Question

I am trying to login to a REST API using HTTP Basic Authentication but it is not working and giving the error

HTTP error 400: Bad Request

Here is my code:

import urllib.parse
import urllib.request
import urllib.response

# create an authorization handler
#auth_handler = urllib.request.HTTPPasswordMgrWithDefaultRealm()
auth_handler = urllib.request.HTTPBasicAuthHandler()


# Add the username and password.
# If we knew the realm, we could use it instead of None.

userName = "username"
passWord  = "pass"
top_level_url = "http URL"
auth_handler.add_password(None, top_level_url, userName,passWord)


# create "opener" (OpenerDirector instance)
opener = urllib.request.build_opener(auth_handler)



# Install the opener.
# Now all calls to urllib.request.urlopen use our opener.
urllib.request.install_opener(opener)

# use the opener to fetch a URL
try:
    result = opener.open(top_level_url)
    #result = urllib.request.urlopen(top_level_url)
    messages = result.read()
    print (messages)  
except IOError as e:
    print (e)

Answer 1

urllib.request.HTTPBasicAuthHandler() by default uses HTTPPasswordMgr. The HTTPPasswordMgr contains a map that has the password from realm and the top_level_url.

When you perform the request and the server returns 401. The returned HTTP headers contains:

Www-Authenticate: Basic realm="a-value"

The HTTPPasswordMgr searches (user, password) for the returned realm and a new request will be sent with (user, password).

When you write:

auth_handler = urllib.request.HTTPBasicAuthHandler()
# Never use None to realm parameter.
auth_handler.add_password(None, top_level_url, userName,passWord)

You expect that the server sends None realm (but it's not possible). If you want your server to send an empty realm in Www-Autheader, you should use

auth_handler.add_password('', top_level_url, userName,passWord)

You can use HTTPPasswordMgrWithDefaultRealm instead of HTTPPasswordMgr to ignore the returned realm:

auth_handler = urllib.request.HTTPBasicAuthHandler(
    urllib.request.HTTPPasswordMgr()
)
auth_handler.add_password(None, top_level_url, userName,passWord))

If your server sends you a 400 response code, with your sample code, then the authentication was not asked.

Answer 2

I would also use requests library as recommended by larsks, it makes HTTP requests so much easier.

That said, here is a working code sample using urllib

import urllib.parse
import urllib.request
import urllib.response

username = "my_username"
password  = "my_password"
top_level_url = "URL"

# create an authorization handler
p = urllib.request.HTTPPasswordMgrWithDefaultRealm()
p.add_password(None, top_level_url, username, password)

auth_handler = urllib.request.HTTPBasicAuthHandler(p)

opener = urllib.request.build_opener(auth_handler)

urllib.request.install_opener(opener)

try:
    result = opener.open(top_level_url)
    messages = result.read()
    print (messages)
except IOError as e:
    print (e)

Another detail - I tried your own code sample and I got back "http 401 unauthorized", which would be the expected response in case of failed or missing auth.

However you claim that you got http 400 bad request, which leads me to think that you either have the wrong url or there is some other issue as well

Answer 3

The requests library offers a far easier way of making this sort of request:

import requests

response = requests.get('http://service.example.com',
                        auth=requests.auth.HTTPBasicAuth(
                          'username',
                          'password'))
print response.text

In my own testing this works out fine, while a solution involving urllib.request (like yours, or using the code verbatim from the examples in the documentation) will fail to send the Authentication: header.

Answer 4

The following python3 code will work:

import urllib.request
import base64
req = urllib.request.Request(download_url)

credentials = ('%s:%s' % (username, password))
encoded_credentials = base64.b64encode(credentials.encode('ascii'))
req.add_header('Authorization', 'Basic %s' % encoded_credentials.decode("ascii"))

with urllib.request.urlopen(req) as response, open(out_file_path, 'wb') as 
out_file:
    data = response.read()
    out_file.write(data)