Difference between list comprehension and generato

Question

What is the difference between list comprehensions and generator comprehensions with yield inside? Both return a generator object (listcomp and genexpr respectively), but upon full evaluation the latter adds what seem to be rather superfluous Nones.

>>> list([(yield from a) for a in zip("abcde", itertools.cycle("12"))])
['a', '1', 'b', '2', 'c', '1', 'd', '2', 'e', '1']

>>> list(((yield from a) for a in zip("abcde", itertools.cycle("12"))))
['a', '1', None, 'b', '2', None, 'c', '1', None, 'd', '2', None, 'e', '1', None]

How come? What is the scientific explanation?

Answer 1

TLDR: A generator expression uses an implicit yield, which returns None from the yield from expression.

There are actually two things behaving differently here. Your list comprehension is actually thrown away...

• Once again with clarity

Understanding this is easiest if you transform the expressions to equivalent functions. For clarity, let's write this out:

listcomp = [<expr> for a in b]
def listfunc():
    result = []
    for a in b:
        result.append(<expr>)
    return result

gencomp = (<expr> for a in b)
def genfunc():
    for a in b:
        yield <expr>

To replicate the initial expressions, the key is to replace <expr> with (yield from a). This is a simple textual replacement:

def listfunc():
    result = []
    for a in b:
        result.append((yield from a))
    return result

def genfunc():
    for a in b:
        yield (yield from a)

With b = ((1,), (2,)), we would expect the output 1, 2. Indeed, both replicate the output of their respective expression/comprehension forms.

As explained elsewhere, yield (yield from a) should make you suspicious. However, result.append((yield from a)) should make you cringe...

• Yielding the answer

Let's look at the generator first. Another rewrite makes it obvious what is going on:

def genfunc():
    for a in b:
        result = (yield from a)
        yield result

For this to be valid, result must have a value - namely None. The generator does not yield the (yield from a) expression, but its result. You only get the content of a as a side effect of evaluating the expression.

• Returning to the question

If you check the type of your "list comprehension", it is not list - it is generator. <listcomp> is just its name. Yes, that's not a Moon, that's a fully functional generator.

Remember how our transformation put a yield from inside a function? Yepp, that is how you define a generator! Here is our function version, this time with print sprinkled on it:

def listfunc():
    result = []
    for a in b:
        result.append((yield from a))
        print(result[-1])
    print(result)
    return result

Evaluating list(listfunc()) prints None, None, and [None, None] and returns [1, 2]. Your actual list contains those None that sneaked into the generator as well! However, it is thrown away and the result is again just a side effect. This is what actually happens:

• A generator is created upon evaluating the list comprehension/listfunc.
• Feeding it to list iterates over it...
  • yield from a provides the values of a and returns None
  • None is stored in the result list

• At the end of the iteration...

  • return raises StopIteration with a value of [None, None]
  • The list constructor ignores this and throws the value away

• Moral of this story

Don't use yield from inside of comprehensions. It does not do what you think it does.

Answer 2

The value of the yield from expression is None. The fact that your second example is a generator expression means that it is already implicitly yielding from the iterator, so it will also yield the value of the yield from expression. See this for a more detailed answer.