Wait on an ordinary function which calls an async

Question

For a project, I want to be able to have simultaneously a sync and async version of a library, the sync version has most logic parts and the async must call the sync version in a async way. For example I have a class which gets an http requester in constructor, this requester handle sync or async internally:

    .
├── async
│   └── foo.py
├── foo.py
└── main.py
└── requester.py

# requester.py
class Requester():
    def connect():
        return self._connect('some_address')

class AsynRequester():
    async def connect():
        return await self._connect('some_address')

# foo.py                                                                                                                                              
class Foo:
    def __init__(self, requester):
        self._requester = requester

    def connect(self):
        self.connect_info = self._requester.connect('host') # in async version it would be called by an await internally

# async/foo.py                                                                                                                                        
from foo import Foo as RawFoo

class Foo(RawFoo):
    async def connect(self):
        return await super(RawFoo, self).connect()

# main.py                                                                                                                                             
from async.foo import Foo # from foo import Foo                                                                                                       
from requester import AsynRequester # from requester import Requester

def main():
    f = Foo(AsyncRequester()) # or Foo(Requester()) where we use sync requester

    await f.connect() # or f.connect() if we are using sync methods    

But async connect finally calls the sync connect of sync class type of Foo (which is parent of async class) which internally calls requester.connect function. It is impossible because requester.connect internally has called await connect when it had being used in async mode but it is calling without any await.

All of my tests have been written for sync version, because async tests is not efficient as they must be, also I must write tests for one version and be sure that both versions would work correctly. How can I have both version at the same time which are using the same logic and just the I/O calls are separated.

Answer 1

the sync version has most logic parts and the async must call the sync version in a async way

It is possible, but it's a lot of work, as you're effectively fighting the function color mismatch. Your logic will have to be written in an async fashion, with some hacks to allow it to work in sync mode.

For example, a logic method would look like this:

# common code that doesn't assume it's either sync or async
class FooRaw:
    async def connect(self):
        self.connect_info = await self._to_async(self._requester.connect(ENDPOINT))

    async def hello_logic(self):
        await self._to_async(self.connect())
        self.sock.write('hello %s %s\n' % (USERNAME, PASSWORD))
        resp = await self._to_async(self.sock.readline())
        assert resp.startswith('OK')
        return resp

When running under asyncio, methods like connect and readline are coroutines, so their return value must be awaited. On the other hand, in blocking code self.connect and sock.readline are sync functions that return concrete values. But await is a syntactic construct which is either present or missing, you cannot switch it off at run-time without duplicating code.

To allow the same code to work in sync and async modes, FooRaw.hello_logic always awaits, leaving it to the _to_async method to wrap the result in an awaitable when running outside asyncio. In async classes _asincify awaits its argument and return the result, it's basically a no-op. In sync classes it returns the received object without awaiting it - but is still defined as async def, so it can be awaited. In that case FooRaw.hello_logic is still a coroutine, but one that never suspends (because the "coroutines" it awaits are all instances of _to_async which doesn't suspend outside asyncio.)

With that in place, the async implementation of hello_logic doesn't need to do anything except choose the right requester and provide the correct _to_async; its connect and hello_logic inherited from FooRaw do the right thing automatically:

class FooAsync(FooRaw):
    def __init__(self):
        self._requester = AsyncRequester()

    @staticmethod
    async def _to_async(x):
        # we're running async, await X and return the result
        result = await x
        return result

The sync version will, in addition to implementing _to_async, need to wrap the logic methods to "run" the coroutine:

class FooSync(FooRaw):
    def __init__(self):
        self._requester = SyncRequester()

    @staticmethod
    async def _to_async(x):
        # we're running sync, X is the result we want
        return x

    # the following can be easily automated by a decorator

    def connect(self):
        return _run_sync(super().connect())

    def hello_logic(self):
        return _run_sync(super().hello_logic())

Note that it is possible to run the coroutine outside the event loop only because the FooSync.hello_logic is a coroutine in name only; the underlying requester uses blocking calls, so FooRaw.connect and others never really suspend, they complete their execution in a single run. (This is similar to a generator that does some work without ever yielding anything.) This property makes the _run_sync helper straightforward:

def _run_sync(coro):
    try:
        # start running the coroutine
        coro.send(None)
    except StopIteration as e:
        # the coroutine has finished; return the result
        # stored in the `StopIteration` exception
        return e.value
    else:
        raise AssertionError("coroutine suspended")