Python closure: Write to variable in parent scope

Question

I have the following code inside a function:

stored_blocks = {}
def replace_blocks(m):
    block = m.group(0)
    block_hash = sha1(block)
    stored_blocks[block_hash] = block
    return '{{{%s}}}' % block_hash

num_converted = 0
def convert_variables(m):
    name = m.group(1)
    num_converted += 1
    return '<%%= %s %%>' % name

fixed = MATCH_DECLARE_NEW.sub('', template)
fixed = MATCH_PYTHON_BLOCK.sub(replace_blocks, fixed)
fixed = MATCH_FORMAT.sub(convert_variables, fixed)

Adding elements to stored_blocks works fine, but I cannot increase num_converted in the second subfunction:

UnboundLocalError: local variable 'num_converted' referenced before assignment

I could use global but global variables are ugly and I really don't need that variable to be global at all.

So I'm curious how I can write to a variable in the parent function's scope. nonlocal num_converted would probably do the job, but I need a solution that works with Python 2.x.

Answer 1

Problem: This is because Python's scoping rules are demented. The presence of the += assignment operator marks the target, num_converted, as local to the enclosing function's scope, and there is no sound way in Python 2.x to access just one scoping level out from there. Only the global keyword can lift variable references out of the current scope, and it takes you straight to the top.

Fix: Turn num_converted into a single-element array.

num_converted = [0]
def convert_variables(m):
    name = m.group(1)
    num_converted[0] += 1
    return '<%%= %s %%>' % name

Answer 2

I have couple of remarks.

First, one application for such nested functions comes up when dealing with raw callbacks, as are used in libraries like xml.parsers.expat. (That the library authors chose this approach may be objectionable, but ... there are reasons to use it nonetheless.)

Second: within a class, there are much nicer alternatives to the array (num_converted[0]). I suppose this is what Sebastjan was talking about.

class MainClass:
    _num_converted = 0
    def outer_method( self ):
        def convert_variables(m):
            name = m.group(1)
            self._num_converted += 1
            return '<%%= %s %%>' % name

It's still odd enough to merit a comment in the code... But the variable is at least local to the class.

Answer 3

(see below for the edited answer)

You can use something like:

def convert_variables(m):
    name = m.group(1)
    convert_variables.num_converted += 1
    return '<%%= %s %%>' % name

convert_variables.num_converted = 0

This way, num_converted works as a C-like "static" variable of the convert_variable method

---

(edited)

def convert_variables(m):
    name = m.group(1)
    convert_variables.num_converted = convert_variables.__dict__.get("num_converted", 0) + 1
    return '<%%= %s %%>' % name

This way, you don't need to initialize the counter in the main procedure.

Answer 4

Modified from: https://stackoverflow.com/a/40690954/819544

You can leverage the inspect module to access the calling scope's globals dict and write into that. That means this trick can even be leveraged to access the calling scope from a nested function defined in an imported submodule.

import inspect 

def get_globals(scope_level=0):
    return dict(inspect.getmembers(inspect.stack()[scope_level][0]))["f_globals"]

num_converted = 0
def foobar():
    get_globals(0)['num_converted'] += 1

foobar()
print(num_converted) 
# 1

Play with the scope_level argument as needed. Setting scope_level=1 works for a function defined in a submodule, scope_level=2 for the inner function defined in a decorator in a submodule, etc.

NB: Just because you can do this, doesn't mean you should.

Answer 5

What about using a class instance to hold the state? You instantiate a class and pass instance methods to subs and those functions would have a reference to self...

Answer 6

Using the global keyword is fine. If you write:

num_converted = 0
def convert_variables(m):
    global num_converted
    name = m.group(1)
    num_converted += 1
    return '<%%= %s %%>' % name

... num_converted doesn't become a "global variable" (i.e. it doesn't become visible in any other unexpected places), it just means it can be modified inside convert_variables. That seems to be exactly what you want.

To put it another way, num_converted is already a global variable. All the global num_converted syntax does is tell Python "inside this function, don't create a local num_converted variable, instead, use the existing global one.