Requests SSL connection timeout

Question

I'm using python requests to send http requests to www.fredmeyer.com

I can't even get past an initial get request to this domain. doing a simple requests.get results in the connection hanging and never timing out. i've verified i have access to this domain and am able to run the request on my local machine. can anyone replicate

Answer 1

The site seems to have some filtering enabled to prohibit bots or similar. The following HTTP request works currently with the site:

GET / HTTP/1.1
Host: www.fredmeyer.com
Connection: keep-alive
Accept: text/html
Accept-Encoding:

If the Connection header is removed or its value changed to close it will hang. If the (empty) Accept-Encoding header is missing it will also hang. If the Accept line is missing it will return 403 Forbidden.

In order to access this site with requests the following currently works for me:

import requests
headers = { 'Accept':'text/html', 'Accept-Encoding': '', 'User-Agent': None }
resp = requests.get('https://www.fredmeyer.com', headers=headers)
print(resp.text)

Note that the heuristics used by the site to detect bots might change, so this might stop working in the future.