Python class scoping rules

Question

EDIT: Looks like this is a very old "bug" or, actually, feature. See, e.g., this mail

I am trying to understand the Python scoping rules. More precisely, I thought that I understand them but then I found this code here:

x = "xtop"
y = "ytop"
def func():
    x = "xlocal"
    y = "ylocal"
    class C:
        print(x)
        print(y)
        y = 1
func()

In Python 3.4 the output is:

xlocal
ytop

If I replace the inner class by a function then it reasonably gives UnboundLocalError. Could you explain me why it behaves this strange way with classes and what is the reason for such choice of scoping rules?

Answer 1

First focus on the case of a closure -- a function within a function:

x = "xtop"
y = "ytop"
def func():
    x = "xlocal"
    y = "ylocal"
    def inner():
 #       global y
        print(x)
        print(y)
        y='inner y'
        print(y)
    inner()  

Note the commented out global in inner If you run this, it replicates the UnboundLocalError you got. Why?

Run dis.dis on it:

>>> import dis
>>> dis.dis(func)
  6           0 LOAD_CONST               1 ('xlocal')
              3 STORE_DEREF              0 (x)

  7           6 LOAD_CONST               2 ('ylocal')
              9 STORE_FAST               0 (y)

  8          12 LOAD_CLOSURE             0 (x)
             15 BUILD_TUPLE              1
             18 LOAD_CONST               3 (<code object inner at 0x101500270, file "Untitled 3.py", line 8>)
             21 LOAD_CONST               4 ('func.<locals>.inner')
             24 MAKE_CLOSURE             0
             27 STORE_FAST               1 (inner)

 14          30 LOAD_FAST                1 (inner)
             33 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
             36 POP_TOP
             37 LOAD_CONST               0 (None)
             40 RETURN_VALUE

Note the different access mode of x vs y inside of func. The use of y='inner y' inside of inner has created the UnboundLocalError

Now uncomment global y inside of inner. Now you have unambiguously create y to be the top global version until resigned as y='inner y'

With global uncommented, prints:

xlocal
ytop
inner y

You can get a more sensible result with:

x = "xtop"
y = "ytop"
def func():
    global y, x
    print(x,y)
    x = "xlocal"
    y = "ylocal"
    def inner():
        global y
        print(x,y)
        y = 'inner y'
        print(x,y)
    inner()    

Prints:

xtop ytop
xlocal ylocal
xlocal inner y

---

The analysis of the closure class is complicated by instance vs class variables and what / when a naked class (with no instance) is being executed.

The bottom line is the same: If you reference a name outside the local namespace and then assign to the same name locally you get a surprising result.

The 'fix' is the same: use the global keyword:

x = "xtop"
y = "ytop"
def func():
    global x, y
    x = "xlocal"
    y = "ylocal"
    class Inner:
        print(x, y)
        y = 'Inner_y'
        print(x, y) 

Prints:

xlocal ylocal
xlocal Inner_y

You can read more about Python 3 scope rules in PEP 3104

Answer 2

TL;DR: This behaviour has existed since Python 2.1 PEP 227: Nested Scopes, and was known back then. If a name is assigned to within a class body (like y), then it is assumed to be a local/global variable; if it is not assigned to (x), then it also can potentially point to a closure cell. The lexical variables do not show up as local/global names to the class body.

---

On Python 3.4, dis.dis(func) shows the following:

>>> dis.dis(func)
  4           0 LOAD_CONST               1 ('xlocal')
              3 STORE_DEREF              0 (x)

  5           6 LOAD_CONST               2 ('ylocal')
              9 STORE_FAST               0 (y)

  6          12 LOAD_BUILD_CLASS
             13 LOAD_CLOSURE             0 (x)
             16 BUILD_TUPLE              1
             19 LOAD_CONST               3 (<code object C at 0x7f083c9bbf60, file "test.py", line 6>)
             22 LOAD_CONST               4 ('C')
             25 MAKE_CLOSURE             0
             28 LOAD_CONST               4 ('C')
             31 CALL_FUNCTION            2 (2 positional, 0 keyword pair)
             34 STORE_FAST               1 (C)
             37 LOAD_CONST               0 (None)
             40 RETURN_VALUE

The LOAD_BUILD_CLASS loads the builtins.__build_class__ on the stack; this is called with arguments __build_class__(func, name); where func is the class body, and name is 'C'. The class body is the constant #3 for the function func:

>>> dis.dis(func.__code__.co_consts[3])
  6           0 LOAD_NAME                0 (__name__)
              3 STORE_NAME               1 (__module__)
              6 LOAD_CONST               0 ('func.<locals>.C')
              9 STORE_NAME               2 (__qualname__)

  7          12 LOAD_NAME                3 (print)
             15 LOAD_CLASSDEREF          0 (x)
             18 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             21 POP_TOP

  8          22 LOAD_NAME                3 (print)
             25 LOAD_NAME                4 (y)
             28 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             31 POP_TOP

  9          32 LOAD_CONST               1 (1)
             35 STORE_NAME               4 (y)
             38 LOAD_CONST               2 (None)
             41 RETURN_VALUE

Within the class body, x is accessed with LOAD_CLASSDEREF (15) while y is load with LOAD_NAME (25). The LOAD_CLASSDEREF is a Python 3.4+ opcode for loading values from closure cells specifically within class bodies (in previous versions, the generic LOAD_DEREF was used); the LOAD_NAME is for loading values from locals and then globals. However closure cells show up neither as locals nor globals.

Now, because the name y is stored to within the class body (35), it is consistently being used as not a closure cell but a local/global name. The closure cells do not show up as local variables to the class body.

This behaviour has been true ever since implementing PEP 227 - nested scopes. And back then BDFL stated that this should not be fixed - and thus it has been for these 13+ years.

---

The only change since PEP 227 is the addition of nonlocal in Python 3; if one uses it within the class body, the class body can set the values of the cells within the containing scope:

x = "xtop"
y = "ytop"
def func():
    x = "xlocal"
    y = "ylocal"
    class C:
        nonlocal y  # y here now refers to the outer variable
        print(x)
        print(y)
        y = 1

    print(y)
    print(C.y)

func()

The output now is

xlocal
ylocal
1
Traceback (most recent call last):
  File "test.py", line 15, in <module>
    func()
  File "test.py", line 13, in func
    print(C.y)
AttributeError: type object 'C' has no attribute 'y'

That is, print(y) read the value of the cell y of the containing scope, and y = 1 set the value in that cell; in this case, no attribute was created for the class C.