Is it wise to use with with statements in generato

Question

Consider the following Python code:

def values():
    with somecontext():
        yield 1
        yield 2
for v in values():
    print(v)
    break

In this case, does Python guarantee that the generator is properly closed and, thus, that the context is exited?

I realize that it, in practice, is going to be the case in CPython due to reference counting and eager destruction of the generator, but does Python guarantee this behavior? I do notice that it does indeed not work in Jython, so should that be considered a bug or allowable behavior?

Answer 1

If your emphasis is on safety, you can always wrap the generator in a contextlib.closing - this seems like the most straightforward solution:

from contextlib import closing

with closing(gen_values()) as values:
    for value in values:
        ...

In fact, if it were me I'd write the function as

def gen_values():
    def inner():
        ...

    return closing(inner())

ensuring that any user has to put this in a with to use it.

Answer 2

Yes, you can use a with statement in a generator without issue. Python will handle the context correctly, because the generator will be closed when garbage collected.

In the generator a GeneratorExit exception is raised when the generator is garbage collected, because it'll be closed at that time:

>>> from contextlib import contextmanager
>>> @contextmanager
... def somecontext():
...     print 'Entering'
...     try:
...         yield None
...     finally:
...         print 'Exiting'
... 
>>> def values():
...     with somecontext():
...         yield 1
...         yield 2
... 
>>> next(values())
Entering
Exiting
1

This is part of PEP 342, where closing a generator raises the exception. Reaping a generator that has no references left should always close that generator, if Jython is not closing the generator I'd consider that a bug.

See points 4 and 5 of the Specification Summary:

4. Add a close() method for generator-iterators, which raises GeneratorExit at the point where the generator was paused. If the generator then raises StopIteration (by exiting normally, or due to already being closed) or GeneratorExit (by not catching the exception), close() returns to its caller. If the generator yields a value, a RuntimeError is raised. If the generator raises any other exception, it is propagated to the caller. close() does nothing if the generator has already exited due to an exception or normal exit.

5. Add support to ensure that close() is called when a generator iterator is garbage-collected.

The only caveat then is that in Jython, IronPython and PyPy the garbage collector is not guaranteed to be run before exiting the interpreter. If this is important to your application you can explicitly close the generator:

gen = values()
next(gen)
gen.close()

or trigger garbage collection explicitly.