SSL error using Python Requests to access Shibbole

Question

I'm trying to access a journal article hosted by an academic service provider (SP), using a Python script.

The server authenticates using a Shibboleth login. I read Logging into SAML/Shibboleth authenticated server using python and tried to implement a login with Python Requests.

The script starts by querying the SP for the link leading to my IDP institution, and is supposed then to authenticate automatically with the IDP. The first part works, but when following the link to the IDP, it chokes on an SSL error. Here is what I used:

import requests
import lxml.html

LOGINLINK = 'https://www.jsave.org/action/showLogin?redirectUri=%2F'
USERAGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0'

s = requests.session()
s.headers.update({'User-Agent' : USERAGENT})

# getting the page where you can search for your IDP
# need to get the cookies so we can continue
response = s.get(LOGINLINK)
rtext = response.text
print('Don\'t see your school?' in rtext) # prints True

# POSTing the name of my institution
data = {
    'institutionName' : 'tubingen',
    'submitForm' : 'Search',
    'currUrl' : '%2Faction%2FshowBasicSearch',
    'redirectUri' : '%2F',
    'activity' : 'isearch'
}
response = s.post(BASEURL + '/action/showLogin', data=data)
rtext = response.text
print('university of tubingen' in rtext) # prints True

# get the link that leads to the IDP
tree = lxml.html.fromstring(rtext)
loginlinks = tree.cssselect('a.extLogin')
if (loginlinks):
    loginlink = loginlinks[0].get('href')
else: 
    exit(1)

print('continuing to IDP')
response = s.get(loginlink)
rtext = response.text
print('zentrale Anmeldeseite' in rtext)

This yields:

continuing to IDP...

2014-04-04 10:04:06,010 - INFO - Starting new HTTPS connection (1): idp.uni-tuebingen.de
Traceback (most recent call last):

File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 480, in urlopen
body=body, headers=headers)

File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 285, in _make_request
conn.request(method, url, **httplib_request_kw)

File "/usr/lib/python3.4/http/client.py", line 1066, in request
self._send_request(method, url, body, headers)

File "/usr/lib/python3.4/http/client.py", line 1104, in _send_request
self.endheaders(body)

File "/usr/lib/python3.4/http/client.py", line 1062, in endheaders
self._send_output(message_body)

File "/usr/lib/python3.4/http/client.py", line 907, in _send_output
self.send(msg)

File "/usr/lib/python3.4/http/client.py", line 842, in send
self.connect()

File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/connection.py", line 164, in connect
ssl_version=resolved_ssl_version)

File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/util.py", line 639, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)

File "/usr/lib/python3.4/ssl.py", line 344, in wrap_socket
_context=self)

File "/usr/lib/python3.4/ssl.py", line 540, in __init__
self.do_handshake()

File "/usr/lib/python3.4/ssl.py", line 767, in do_handshake
self._sslobj.do_handshake()

ssl.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:598)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/lib/python3.4/site-packages/requests/adapters.py", line 330, in send
timeout=timeout

File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 504, in urlopen
raise SSLError(e)

requests.packages.urllib3.exceptions.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:598)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./try.py", line 154, in <module>
response = s.get(loginlink)

File "/usr/lib/python3.4/site-packages/requests/sessions.py", line 395, in get
return self.request('GET', url, **kwargs)

File "/usr/lib/python3.4/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)

File "/usr/lib/python3.4/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)

File "/usr/lib/python3.4/site-packages/requests/adapters.py", line 385, in send
raise SSLError(e)

requests.exceptions.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:598)

Using s.get(loginlink, verify=False) yields exactly the same error. Simply using urllib.request.urlopen(loginlink) does so, too.

Printing and pasting the link into Firefox, on the other hand, works fine.

Answer 1

After trying with openssl s_client it looks like the destination idp.uni-tuebingen.de:443 is only support SSLv3 and misbehaving on anything newer. With forcing SSLv3 one gets:

$ openssl s_client -connect idp.uni-tuebingen.de:443 -ssl3
CONNECTED(00000003)
depth=3 C = DE, O = Deutsche Telekom AG, OU = T-TeleSec Trust Center, CN = Deutsche Telekom Root CA 2
...

But with default setup or forcing TLv1 (-tls1) it only returns an alert:

openssl s_client -connect idp.uni-tuebingen.de:443 
CONNECTED(00000003)
140493591938752:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:741:

So you need to find a way to force SSLv3 for this connection. I'm not familiar with the python at this point but maybe http://docs.python-requests.org/en/latest/user/advanced/ chapter "Example: Specific SSL Version" helps.

And why it works with firefox: the browsers usually retry with a downgraded SSL version if the connects with the safer versions fail. E.g. everybody is trying to work around broken stuff so that the owner of the broken stuff has no intention to fix it :(