How does this lambda/yield/generator comprehension

Question

I was looking through my codebase today and found this:

def optionsToArgs(options, separator='='):
    kvs = [
        (
            "%(option)s%(separator)s%(value)s" %  
            {'option' : str(k), 'separator' : separator, 'value' : str(v)}
        ) for k, v in options.items()
    ]
    return list(
        reversed(
            list(
                    (lambda l, t: 
                        (lambda f: 
                            (f((yield x)) for x in l)
                        )(lambda _: t)
                    )(kvs, '-o')
                )
            )
        )

It seems to take a dict of parameters and turn them into a list of parameters for a shell command. It looks like it's using yield inside a generator comprehension, which I thought would be impossible...?

>>> optionsToArgs({"x":1,"y":2,"z":3})
['-o', 'z=3', '-o', 'x=1', '-o', 'y=2']

How does it work?

Answer 1

Since Python 2.5, yield <value> is an expression, not a statement. See PEP 342.

The code is hideously and unnecessarily ugly, but it's legal. Its central trick is using f((yield x)) inside the generator expression. Here's a simpler example of how this works:

>>> def f(val):
...     return "Hi"
>>> x = [1, 2, 3]
>>> list(f((yield a)) for a in x)
[1, 'Hi', 2, 'Hi', 3, 'Hi']

Basically, using yield in the generator expression causes it to produce two values for every value in the source iterable. As the generator expression iterates over the list of strings, on each iteration, the yield x first yields a string from the list. The target expression of the genexp is f((yield x)), so for every value in the list, the "result" of the generator expression is the value of f((yield x)). But f just ignores its argument and always returns the option string "-o". So on every step through the generator, it yields first the key-value string (e.g., "x=1"), then "-o". The outer list(reversed(list(...))) just makes a list out of this generator and then reverses it so that the "-o"s will come before each option instead of after.

However, there is no reason to do it this way. There are a number of much more readable alternatives. Perhaps the most explicit is simply:

kvs = [...] # same list comprehension can be used for this part
result = []
for keyval in kvs:
   result.append("-o")
   result.append(keyval)
return result

Even if you like terse, "clever" code, you could still just do

return sum([["-o", keyval] for keyval in kvs], [])

The kvs list comprehension itself is a bizarre mix of attempted readability and unreadability. It is more simply written:

kvs = [str(optName) + separator + str(optValue) for optName, optValue in options.items()]

You should consider arranging an "intervention" for whoever put this in your codebase.

Answer 2

Oh god. Basically, it boils down to this,:

def f(_):              # I'm the lambda _: t
    return '-o'

def thegenerator():   # I'm (f((yield x)) for x in l)
    for x in kvs:
        yield f((yield x))

So when iterated over, thegenerator yields x (a member of kvs) and then the return value of f, which is always -o, all in one iteration over kvs. Whatever yield x returns and what gets passed to f is ignored.

Equivalents:

def thegenerator():   # I'm (f((yield x)) for x in l)
    for x in kvs:
        whatever = (yield x)
        yield f(whatever)

def thegenerator():   # I'm (f((yield x)) for x in l)
    for x in kvs:
        yield x
        yield f(None)

def thegenerator():   # I'm (f((yield x)) for x in l)
    for x in kvs:
        yield x
        yield '-o'

There are lots of ways to do this much simpler, of course. Even with the original double-yield trick, the entire thing could've been

return list(((lambda _: '-o')((yield x)) for x in kvs))[::-1]