HTTPS request via urllib2 fails behind NTLM proxy

Question

Via Python's urllib2 I try to get data over HTTPS while I am behind a corporate NTLM proxy.

I run

proxy_url = ('http://user:pw@ntlmproxy:port/')
proxy_handler = urllib2.ProxyHandler({'http': proxy_url})
opener = urllib2.build_opener(proxy_handler, urllib2.HTTPHandler)
urllib2.install_opener(opener)

f = urllib2.urlopen('https://httpbin.org/ip')
myfile = f.read()
print myfile

but I get as error

urllib2.URLError: <urlopen error [Errno 8] _ssl.c:507: 
EOF occurred in violation of protocol>

How can I fix this error?

Note 0: With the same code I can retrieve the unsecured HTTP equivalent http://httpbin.org/ip.

Note 1: From a normal browser I can access https://httpbin.org/ip (and other HTTPS sites) via the same corporate proxy.

Note 2: I was reading about many similar issues on the net and some suggested that it might be related to certificate verification, but urllib2 does not verify certificates anyway.

Note 3: Some people suggested in similar situations monekypatching, but I guess, there is no way to monkeypatch _ssl.c.

Answer 1

The problem is that Python's standard HTTP libraries do not speak Microsoft's proprietary NTLM authentication protocol fully.

I solved this problem by setting up a local NTLM-capable proxy - ntlmaps did the trick for me.(*) - which providesthe authentication against the corporate proxy and point my python code to this local proxy without authentication credentials.

Additionally I had to add in the above listed python code a proxy_handler for HTTPS. So I replaced the two lines

proxy_url = 'http://user:pw@ntlmproxy:port/' 
proxy_handler = urllib2.ProxyHandler({'http': proxy_url})

with the two lines

proxy_url = 'http://localproxy:localport/' 
proxy_url_https = 'https://localproxy:localport/' 
proxy_handler = urllib2.ProxyHandler({'http': proxy_url, 'https': proxy_url_https})

Then the request works perfectly.

---

(*) ntlmaps is a Python program. Due to some reasons in my personal environment it was for me necessary that the proxy is a python program.)