How to get results out of a Python exec()/

Question

I want to write a tool in Python to prepare a simulation study by creating for each simulation run a folder and a configuration file with some run-specific parameters.

study/
  study.conf
  run1
    run.conf
  run2
    run.conf

The tool should read the overall study configuration from a file including (1) static parameters (key-value pairs), (2) lists for iteration parameters, and (3) some small code snippets to calculate further parameters from the previous ones. The latter are run specific depending on the permutation of the iteration parameters used.

Before writing the run.conf files from a template, I need to run some code like this to determine the specific key-value pairs from the code snippets for that run

code = compile(code_str, 'foo.py', 'exec')
rv=eval(code, context, { })

However, as this is confirmed by the Python documentation, this just leads to a None as return value.

The code string and context dictionary in the example are filled elsewhere. For this discussion, this snippet should do it:

code_str="""import math
math.sqrt(width**2 + height**2)
"""

context = {
    'width' : 30,
    'height' : 10
}

I have done this before in Perl and Java+JavaScript. There, you just give the code snippet to some evaluation function or script engine and get in return a value (object) from the last executed statement -- not a big issue.

Now, in Python I struggle with the fact that eval() is too narrow just allowing one statement and exec() doesn't return values in general. I need to import modules and sometimes do some slightly more complex calculations, e.g., 5 lines of code.

Isn't there a better solution that I don't see at the moment?

During my research, I found some very good discussions about Pyhton eval() and exec() and also some tricky solutions to circumvent the issue by going via the stdout and parsing the return value from there. The latter would do it, but is not very nice and already 5 years old.

Answer 1

Since this whole exec() / eval() thing in Python is a bit weird ... I have chose to re-design the whole thing based on a proposal in the comments to my question (thanks @jonrsharpe).

Now, the whole study specification is a .py module that the user can edit. From there, the configuration setup is directly written to a central object of the whole package. On tool runs, the configuration module is imported using the code below

import imp

# import the configuration as a module
(path, name) = os.path.split(filename)
(name, _) = os.path.splitext(name)
(file, filename, data) = imp.find_module(name, [path])

try:
    module = imp.load_module(name, file, filename, data)
except ImportError as e:
    print(e)
    sys.exit(1)
finally:
    file.close()

Answer 2

The exec function will modify the global parameter (dict) passed to it. So you can use the code below

code_str="""import math
Result1 = math.sqrt(width**2 + height**2)
"""
context = {
    'width' : 30,
    'height' : 10
}

exec(code_str, context)

print (context['Result1']) # 31.6

Every variable code_str created will end up with a key:value pair in the context dictionary. So the dict is the "object" like you mentioned in JavaScript.

Edit1:

If you only need the result of the last line in code_str and try to prevent something like Result1=..., try the below code

code_str="""import math
math.sqrt(width**2 + height**2)
"""

context = { 'width' : 30, 'height' : 10 }

lines = [l for l in code_str.split('\n') if l.strip()]
lines[-1] = '__myresult__='+lines[-1] 

exec('\n'.join(lines), context)
print (context['__myresult__'])

This approach is not as robust as the former one, but should work for most case. If you need to manipulate the code in a sophisticated way, please take a look at the Abstract Syntax Trees

Answer 3

I came across similar needs, and finally figured out a approach by playing with ast:

import ast
code = """
def tf(n):
  return n*n
r=tf(3)
{"vvv": tf(5)}
"""
ast_ = ast.parse(code, '<code>', 'exec')
final_expr = None
for field_ in ast.iter_fields(ast_):
    if 'body' != field_[0]: continue
    if len(field_[1]) > 0 and isinstance(field_[1][-1], ast.Expr):
        final_expr = ast.Expression()
        final_expr.body = field_[1].pop().value
ld = {}
rv = None
exec(compile(ast_, '<code>', 'exec'), None, ld)
if final_expr:
    rv = eval(compile(final_expr, '<code>', 'eval'), None, ld)
print('got locals: {}'.format(ld))
print('got return: {}'.format(rv))

It'll eval instead of exec the last clause if it's an expression, or have all execed and return None.

Output:

got locals: {'tf': <function tf at 0x10103a268>, 'r': 9}
got return: {'vvv': 25}