C Python: Running Python code within a context

Question

The Python C API function PyEval_EvalCode let's you execute compiled Python code. I want to execute a block of Python code as if it were executing within the scope of a function, so that it has its own dictionary of local variables which don't affect the global state.

This seems easy enough to do, since PyEval_EvalCode lets you provide a Global and Local dictionary:

PyObject* PyEval_EvalCode(PyCodeObject *co, PyObject *globals, PyObject *locals)

The problem I run into has to do with how Python looks up variable names. Consider the following code, that I execute with PyEval_EvalCode:

myvar = 300
def func():
    return myvar

func()

This simple code actually raises an error, because Python is unable to find the variable myvar from within func. Even though myvar is in the local dictionary in the outer scope, Python doesn't copy it into the local dictionary in the inner scope. The reason for this is as follows:

Whenever Python looks up a variable name, first it checks locals, then it checks globals, and finally it checks builtins. At module scope, locals and globals are the SAME dictionary object. So the statement x = 5 at module scope will place x in the the locals dictionary, which is also the globals dictionary. Now, a function defined at module scope which needs to lookup x won't find x within the function-scope locals, because Python doesn't copy module-scope locals into function-scope locals. But this normally isn't a problem, because it can find x in globals.

x = 5
def foo():
   print(x) # This works because 'x' in globals() == True

It's only with nested functions, that Python seems to copy outer-scope locals into inner-scope locals. (It also seems to do so lazily, only if they are needed within the inner scope.)

def foo():
   x = 5
   def bar():
      print(x) # Now 'x' in locals() == True
   bar()

So the result of all this is that, when executing code at module scope, you HAVE to make sure that your global dictionary and local dictionary are the SAME object, otherwise module-scope functions won't be able to access module-scope variables.

But in my case, I don't WANT the global dictionary and local dictionary to be the same. So I need some way to tell the Python interpreter that I am executing code at function scope. Is there some way to do this? I looked at the PyCompileFlags as well as the additional arguments to PyEval_EvalCodeEx and can't find any way to do this.

Answer 1

Python doesn't actually copy outer-scope locals into inner-scope locals; the documentation for locals states:

Free variables are returned by locals() when it is called in function blocks, but not in class blocks.

Here "free" variables refers to variables closed over by a nested function. It's an important distinction.

The simplest fix for your situation is just to pass the same dict object as globals and locals:

code = """
myvar = 300
def func():
    return myvar

func()
"""
d = {}
eval(compile(code, "<str>", "exec"), d, d)

Otherwise, you can wrap your code in a function and extract it from the compiled object:

s = 'def outer():\n    ' + '\n    '.join(code.strip().split('\n'))
exec(compile(s, '<str>', 'exec').co_consts[0], {}, {})