I want to implement an awaitable and noticed that __await__ 'needs' to be a generator.
From PEP-492:
An object with an __await__ method returning an iterator.
...
Objects with __await__ method are called Future-like objects in the rest of this PEP.
It is a TypeError if __await__ returns anything but an iterator.
In my experience, before await was a statement, yield from was used together with coroutines implemented as generators. Nowadays python (I'm using 3.5) has asynchronous methods using the async def syntax. I therefore consider the yield from syntax as old/deprecated.
So I broke out the interpreter to see how/if this works:
>>> class A:
... def __await__(self):
... yield from (asyncio.sleep(1).__await__())
... return 'spam'
...
>>> a = A()
>>> loop = asyncio.get_event_loop()
>>> loop.run_until_complete(a)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python3.5/asyncio/base_events.py", line 467, in run_until_complete
return future.result()
File "/usr/lib64/python3.5/asyncio/futures.py", line 294, in result
raise self._exception
File "/usr/lib64/python3.5/asyncio/tasks.py", line 240, in _step
result = coro.send(None)
File "/usr/lib64/python3.5/asyncio/tasks.py", line 585, in _wrap_awaitable
return (yield from awaitable.__await__())
File "<stdin>", line 3, in __await__
AttributeError: 'generator' object has no attribute '__await__'
So it appears asyncio.sleep doesn't have the __await__ method. It also feels very awkward to use this yield from syntax.
So I decided to try with the async syntax, just to see if it would work:
>>> class A:
... async def __await__(self):
... await asyncio.sleep(1)
... return 'spam'
...
>>> a = A()
>>>
>>> loop = asyncio.get_event_loop()
>>> loop.run_until_complete(a)
'spam'
It actually seems to work! So now I'm wondering, does the __await__ method really need to be a generator using the yield from syntax?
Edit: When adding a level of indirection, so the awaitable is used in an await statement the problem becomes apparent:
>>> async def test():
... return await A()
...
>>> loop.run_until_complete(test())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "<stdin>", line 2, in test
TypeError: __await__() returned a coroutine
It actually thus needs to be returning a generator like so:
class A:
def __await__(self):
yield from asyncio.sleep(1)
return 'spam'
So it appears asyncio.sleep doesn't have the __await__ method
True, but it doesn't have to have one to be awaitable. The documentation says that __await__, if present, needs to return an iterator, not that await will only work on objects that define __await__. In fact, it explicitly documents that the argument to await can be one of:
A native coroutine object returned from a native coroutine function.
A generator-based coroutine object returned from a function decorated with types.coroutine().
An object with an __await__ method returning an iterator.
An object defined in C providing the Python/C equivalent of the __await__ special method.
So now I'm wondering, does the __await__ method really need to be a generator using the yield from syntax?
If you actually have an __await__ method, it does need to return an iterator.
In order to work in an await expression, __await__ does not need to be a generator. However, certain operations are only available if the result of __await__ support the generator interface.
Namely, it is not possible to send values or throw exceptions into an iterator-__await__. Only None can be "sent" to an iterator-__await__, as if generator.__next__ were used.
Let's consider a simple Awaitable that returns an iterator from its __await__.
class Iter:
"""Basic iterator that yields the same value"""
def __next__(self): return 1
def __iter__(self): return self
class IterAwait:
"""Awaitable that uses an iterator for __await__"""
def __await__(self):
return Iter()
We can check that they implement the desired interfaces:
>>> from collections.abc import Awaitable, Iterator, Generator
>>> isinstance(IterAwait(), Awaitable)
True
>>> isinstance(IterAwait().__await__(), Iterator)
True
>>> isinstance(IterAwait().__await__(), Generator)
False
In order to see how this interacts with await, we wrap it in a coroutine:
async def iter_await():
await IterAwait()
Every operation we perform on iter_await with the full coroutine/generator interface is forwarded by await to our iterator-__await__. This allows to study how the iterator-__await__ receives signals:
>>> test_iter = iter_await()
>>> test_iter.send(3) # 0. does it appear like a coroutine?
TypeError: can`t send non-None value to a just-started coroutine
>>> test_iter.send(None) # 1. must initialise just-started coroutine
1
>>> test_iter.send(None) # 2. repeatedly use the underlying iterator
1
>>> next(test_iter) # 3. do we expose the iterator?
TypeError: 'coroutine' object is not an iterator
>>> test_iter.send(3) # 4. can we send non-None values?
AttributeError: 'Iter' object has no attribute 'send'
>>> test_iter = iter_await() # we just broke the coroutine...
>>> test_iter.send(None) # ...need a new one
1
>>> test_iter.throw(KeyError) # 4. can we throw Exceptions?
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in iter_await
KeyError
As can be seen, await can handle an iterator-__await__, but does not forward all operations. However, some are translated and some are handled early.
- It is always possible to
.send(None), which is translated to a bare __next__(). (1, 2)
- The coroutine does not magically expose
.__next__ (3) and cannot translate .send with a value either (4).
- It is possible to
.throw an exception, but await handles it early in the coroutine.
Note that await uses the throw and send method as available. If the result of __await__ implements send but not throw or vice versa, the functionality present is used. Only __next__ is mandatory.
