How do lexical closures work?

Question

While I was investigating a problem I had with lexical closures in Javascript code, I came along this problem in Python:

flist = []

for i in xrange(3):
    def func(x): return x * i
    flist.append(func)

for f in flist:
    print f(2)

Note that this example mindfully avoids lambda. It prints "4 4 4", which is surprising. I'd expect "0 2 4".

This equivalent Perl code does it right:

my @flist = ();

foreach my $i (0 .. 2)
{
    push(@flist, sub {$i * $_[0]});
}

foreach my $f (@flist)
{
    print $f->(2), "\n";
}

"0 2 4" is printed.

Can you please explain the difference ?

---

Update:

The problem is not with i being global. This displays the same behavior:

flist = []

def outer():
    for i in xrange(3):
        def inner(x): return x * i
        flist.append(inner)

outer()
#~ print i   # commented because it causes an error

for f in flist:
    print f(2)

As the commented line shows, i is unknown at that point. Still, it prints "4 4 4".

Answer 1

What is happening is that the variable i is captured, and the functions are returning the value it is bound to at the time it is called. In functional languages this kind of situation never arises, as i wouldn't be rebound. However with python, and also as you've seen with lisp, this is no longer true.

The difference with your scheme example is to do with the semantics of the do loop. Scheme is effectively creating a new i variable each time through the loop, rather than reusing an existing i binding as with the other languages. If you use a different variable created external to the loop and mutate it, you'll see the same behaviour in scheme. Try replacing your loop with:

(let ((ii 1)) (
  (do ((i 1 (+ 1 i)))
      ((>= i 4))
    (set! flist 
      (cons (lambda (x) (* ii x)) flist))
    (set! ii i))
))

Take a look here for some further discussion of this.

[Edit] Possibly a better way to describe it is to think of the do loop as a macro which performs the following steps:

1. Define a lambda taking a single parameter (i), with a body defined by the body of the loop,
2. An immediate call of that lambda with appropriate values of i as its parameter.

ie. the equivalent to the below python:

flist = []

def loop_body(i):      # extract body of the for loop to function
    def func(x): return x*i
    flist.append(func)

map(loop_body, xrange(3))  # for i in xrange(3): body

The i is no longer the one from the parent scope but a brand new variable in its own scope (ie. the parameter to the lambda) and so you get the behaviour you observe. Python doesn't have this implicit new scope, so the body of the for loop just shares the i variable.

Answer 2

Here's how you do it using the functools library (which I'm not sure was available at the time the question was posed).

from functools import partial

flist = []

def func(i, x): return x * i

for i in xrange(3):
    flist.append(partial(func, i))

for f in flist:
    print f(2)

Outputs 0 2 4, as expected.

Answer 3

look at this:

for f in flist:
    print f.func_closure


(<cell at 0x00C980B0: int object at 0x009864B4>,)
(<cell at 0x00C980B0: int object at 0x009864B4>,)
(<cell at 0x00C980B0: int object at 0x009864B4>,)

It means they all point to the same i variable instance, which will have a value of 2 once the loop is over.

A readable solution:

for i in xrange(3):
        def ffunc(i):
            def func(x): return x * i
            return func
        flist.append(ffunc(i))