Weird behavior: Lambda inside list comprehension

Question

In python 2.6:

[x() for x in [lambda: m for m in [1,2,3]]]

results in:

[3, 3, 3]

I would expect the output to be [1, 2, 3]. I get the exact same problem even with a non list comprehension approach. And even after I copy m into a different variable.

What am I missing?

Answer 1

I noticed that too. I concluded that lambda are created only once. So in fact your inner list comprehension will give 3 indentical functions all related to the last value of m.

Try it and check the id() of the elements.

[Note: this answer is not correct; see the comments]

Answer 2

Long story short, you don't want to do this. More specifically, what you're encountering is an order of operations problem. You're creating three separate lambda's that all return m, but none of them are called immediately. Then, when you get to the outer list comprehension and they're all called the residual value of m is 3, the last value of the inner list comprehension.

-- For comments --

>>> [lambda: m for m in range(3)]
[<function <lambda> at 0x021EA230>, <function <lambda> at 0x021EA1F0>, <function <lambda> at 0x021EA270>]

Those are three separate lambdas.

And, as further evidence:

>>> [id(m) for m in [lambda: m for m in range(3)]]
[35563248, 35563184, 35563312]

Again, three separate IDs.

Answer 3

Look at the __closure__ of the functions. All 3 point to the same cell object, which keeps a reference to m from the outer scope:

>>> print(*[x.__closure__[0] for x in [lambda: m for m in [1,2,3]]], sep='\n')
<cell at 0x00D17610: int object at 0x1E2139A8>
<cell at 0x00D17610: int object at 0x1E2139A8>
<cell at 0x00D17610: int object at 0x1E2139A8>

If you don't want your functions to take m as a keyword argument, as per unubtu's answer, you could instead use an additional lambda to evaluate m at each iteration:

>>> [x() for x in [(lambda x: lambda: x)(m) for m in [1,2,3]]]
[1, 2, 3]

Answer 4

The effect you’re encountering is called closures, when you define a function that references non-local variables, the function retains a reference to the variable, rather than getting its own copy. To illustrate, I’ll expand your code into an equivalent version without comprehensions or lambdas.

inner_list = []
for m in [1, 2, 3]:
    def Lambda():
         return m
    inner_list.append(Lambda)

So, at this point, inner_list has three functions in it, and each function, when called, will return the value of m. But the salient point is that they all see the very same m, even though m is changing, they never look at it until called much later.

outer_list = []
for x in inner_list:
    outer_list.append(x())

In particular, since the inner list is constructed completely before the outer list starts getting built, m has already reached its last value of 3, and all three functions see that same value.

Answer 5

To make the lambdas remember the value of m, you could use an argument with a default value:

[x() for x in [lambda m=m: m for m in [1,2,3]]]
# [1, 2, 3]

This works because default values are set once, at definition time. Each lambda now uses its own default value of m instead of looking for m's value in an outer scope at lambda execution time.

Answer 6

Personally, I find this a more elegant solution. Lambda returns a function, so if we want to use the function, then we should use it. It's confusing to use the same symbol for the 'anonymous' variable in the lambda and for the generator, so in my example I use a different symbol to make it hopefully more clear.

>>> [ (lambda a:a)(i) for i in range(3)]
[0, 1, 2]
>>> 

it's faster too.

>>> timeit.timeit('[(lambda a:a)(i) for i in range(10000)]',number=10000)
9.231263160705566
>>> timeit.timeit('[lambda a=i:a  for i in range(10000)]',number=10000)
11.117988109588623
>>> 

but not as fast as map:

>>> timeit.timeit('map(lambda a:a,  range(10000))',number=10000)
5.746963977813721

(I ran these tests more than once, result was the same, this was done in python 2.7, results are different in python 3: the two list comprehensions are much closer in performance and both a lot slower, map remains much faster. )