asynchronous aiohttp requests fails, but synchrono

Question

With the following code I get Cannot connect to host ...:443 ssl:True when I use the asynchronous aiohttp. When I use synchronous requests, it succeeds.

The whitehouse.gov links fail, but the google.com succeeds for both async and sync cases.

What is going wrong? This is with python 3.4.2 on FreeBSD8, aiohttp 0.14.4, requests 2.5.3

import asyncio
import aiohttp
import requests

urls = [
    'http://www.whitehouse.gov/cea/', 
    'http://www.whitehouse.gov/omb', 
    'http://www.google.com']


def test_sync():
    for url in urls:
        r = requests.get(url)
        print(r.status_code)


def test_async():
    for url in urls:
        try:
            r = yield from aiohttp.request('get', url)
        except aiohttp.errors.ClientOSError as e:
            print('bad eternal link %s: %s' % (url, e))
        else:
            print(r.status)


if __name__ == '__main__':
    print('async')
    asyncio.get_event_loop().run_until_complete(test_async())
    print('sync')
    test_sync()

The output from this is:

async
bad eternal link http://www.whitehouse.gov/cea: Cannot connect to host www.whitehouse.gov:443 ssl:True
bad eternal link http://www.whitehouse.gov/omb: Cannot connect to host www.whitehouse.gov:443 ssl:True
200
sync
200
200
200

Answer 1

Don't use this answere as it might be equal to disabling certificate checks.

As pointed out by Le Hibou in the comments, ssl.Purpose.CLIENT_AUTH is meant for authenticating clients on the server side.

This value indicates that the context may be used to authenticate Web clients (therefore, it will be used to create server-side sockets).

Looking at the code for create_default_context() shows that certificate checks might be disabled or at least optional in this case.

---

I had the same error message (on Windows) and solved it with the following:

import aiohttp
import ssl


client = aiohttp.ClientSession()
client.post(
    'https://some.foo/bar/',
    json={"foo": "bar"},
    ssl=ssl.create_default_context(purpose=ssl.Purpose.CLIENT_AUTH))

This is using the ssl argument of request() to set a different SSL context than the default. The default is ssl.create_default_context().

The problem is that the default value for ssl.create_default_context() are:

ssl.create_default_context(purpose=Purpose.SERVER_AUTH, cafile=None, capath=None, cadata=None)

The Purpose.SERVER_AUTH doesn't seem to work when validating server certificates on the client side. Instead use Purpose.CLIENT_AUTH when validating as a client.

Answer 2

I suspect certificate validation chain is broken on your machine. On Ubuntu everything is working, as @dano mentioned.

Anyway, you may disable ssl validation by creating custom Connector instance:

import asyncio
import aiohttp

urls = [
    'http://www.whitehouse.gov/cea/',
    'http://www.whitehouse.gov/omb',
    'http://www.google.com']


def test_async():
    connector = aiohttp.TCPConnector(verify_ssl=False)
    for url in urls:
        try:
            r = yield from aiohttp.request('get', url, connector=connector)
        except aiohttp.errors.ClientOSError as e:
            print('bad eternal link %s: %s' % (url, e))
        else:
            print(r.status)


if __name__ == '__main__':
    print('async')
    asyncio.get_event_loop().run_until_complete(test_async())

BTW, requests library is shipped with own certificate bundle. Maybe we need to do the same for aiohttp?

UPD. See also https://github.com/aio-libs/aiohttp/issues/341

Answer 3

I had the same problem on an old Linux server with out of date CA root certificates, and loading certifi CA certificate bundle in a SSLContext fixed the issue.

import aiohttp
import ssl
import certifi

ssl_context = ssl.create_default_context(cafile=certifi.where())
async with aiohttp.ClientSession() as session:
    async with session.get('https://some.foo/bar/', ssl=ssl_context) as response:
        print(await response.text())